Auto-ARGUE: LLM-Based Report Generation Evaluation

Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval 2026 Open access

Read full text
Authors: William Gantt Walden, Marc A. Mason, Orion Weller, Laura Dietz, John M. Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, Dawn Lawrie, James Mayfield
+1 more Eugene Yang
DOI: 10.1145/3805712.3808384
Cited by: 1
References: 15

Abstract

Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools designed for report generation are lacking. Accordingly, we introduce Auto-ARGUE, a robust LLM-based implementation of the recently proposed ARGUE framework for report generation evaluation. We present analysis of Auto-ARGUE on the report generation pilot task from the TREC 2024 NeuCLIR track and on two tasks from the TREC 2024 RAG track, showing good system-level correlations with human judgments. Additionally, we release ARGUE-viz, a web app for visualization and fine-grained analysis of Auto-ARGUE judgments and scores1.

Fields of Study & Subjects

Advanced Text Analysis Techniques Semantic Web and Ontologies

Publication details

Journal / source
Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Pages
5235–5239
Publication date
10 July 2026
Language
EN