Auto-Judge: A Cross-Task Benchmark for Comparing LLM Judges for Citation-Grounded RAG Systems

Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval 2026 Open access

Read full text
Authors: Naghmeh Farzi, Tim Hagen, Eugene Yang, Maik Fröbe, Ronak Pradeep, Hossein A. Rahmani, Xi Wang, Oleg Zendel, Martin Potthast, Laura Dietz
DOI: 10.1145/3805712.3808601
Cited by: 1
References: 37

Abstract

We present the Auto-Judge resource for the meta-evaluation of automated LLM judges, especially judges that evaluate Retrieval-Augmented Generation (RAG) systems that ground their response with citations. The resource couples (i) a data release of topics, pooled RAG responses, and human judgments, with (ii) a standardized protocol and software infrastructure for implementing "LLM-as-a-judge" methods in a reproducible and extensible way, including support for parameter sweeps and variant tracking.

Publication details

Journal / source
Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Pages
3159–3166
Publication date
15 July 2026
Language
EN