Result Assessment Tool (RAT): An Open-Source Toolkit for Conducting Studies based on Search Results

The Result Assessment Tool (RAT) is an open-source software toolkit for conducting research with resultsfrom commercial search engines and other web-based information retrieval systems. Evaluating these “blackbox” systems poses a fundamental challenge for researchers. Traditional evaluation paradigms are often notapplicable, and providers offer minimal, if any, access to their data. In addition, commercial APIs are oftencostly, subject to strict limitations, and may not deliver the same full SERP features (such as AI-generatedoverviews) that end users see, making reliable comparative studies difficult. RAT fills this gap by providing a flexible, open-source, and integrated platform that standardizes the entireresearch workflow. Its modular architecture, consisting of a Flask-based frontend and a Python backend,manages study design, data collection, manual evaluation, and automated analysis. The RAT Frontend is a Flask-based GUI that provides two distinct interfaces for its user groups. The ResearcherView acts as the central control panel for designing studies, allowing researchers to define parameters, managequery lists (including using an integratedQuery Sampler tool), create custom questionnaires, and monitor realtime statistics via an Analyzer module. The Evaluation View is the dedicated interface for study participants(jurors). To ensure reproducibility, jurors are presented with archived screenshots of search results alongsidethe questionnaire, guaranteeing that all participants evaluate the exact same content as it was captured at thetime of collection. The RAT Backend handles the computationally intensive tasks. A job management system controls the twostep data collection process: first, the Search Engine Scraper collects metadata (titles, URLs, snippets) fromSERPs or AI-generated content. Second, the Source Scraper retrieves and archives the full HTML content anda screenshot of each result URL, using Selenium WebDriver to interact with dynamic pages and capture theirfull content. The software is designed for extensibility, so researchers can add their own automated classifiers (e.g., anintegrated classifier to calculate the probability of search engine optimization on a search result) or contributenew modules as RAT extensions.The tool’s effectiveness is shown by its use in various research areas, including in information science toreview the diversity of sources in over 140,000 results, and in political science to research the use of SEO bypolitical candidates. RAT provides a complete, sustainable infrastructure, unlike simple scraping libraries that only provide components. While commercial marketing tools hide their methods, the software offers full methodological transparency. And in contrast to prior single-use academic tools, RAT is built for long-term maintenance andextensibility. The software is available as open-source in a dedicated GitHub repository following the FAIR principles. Wehave additionally made various research data created with RAT freely available. We aim to reach a widerresearch community to carry out important studies on the complicated information systems that influenceour world.