Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents

Evaluating deep research agents that iteratively search the web, analyze information, and generate reports remains a major challenge, especially in assessing long reports and providing fine-grained feedback. To address these gaps, we introduce Deep Research Comparator, a holistic annotation platform for the human evaluation of deep research agents. Our platform displays the generated reports and intermediate steps from two agents side-by-side, and allows annotators to state their preference between two final reports, and provide fine-grained feedback on specific text spans within the report or intermediate steps for each agent separately. Furthermore, we develop Simple Deepresearch, an agent scaffold that serves as a baseline to facilitate the integration of various large language models to transform them into deep research agents for evaluation. Experiment results of three agents with 17 annotators demonstrated strong alignment between human annotations and static benchmarks, while revealing divergence in fine-grained metrics, which highlights the distinct value of fine-grained annotations in providing robust evaluation and actionable data for nuanced agent optimization. We will provide a web-based implementation of our platform that conference attendees can interact with in real-time.