arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation

Literature review tables are essential for summarizing and comparing collections of scientific papers.In this paper, we study automatic generation of such tables from a pool of papers to satisfy a user's information need.Building on recent work (Newman et al., 2024), we move beyond oracle settings by (i) simulating wellspecified yet schema-agnostic user demands that avoid leaking gold column names or values, (ii) explicitly modeling retrieval noise via semantically related but out-of-scope distractor papers verified by human annotators, and (iii) introducing a lightweight, annotation-free, utilization-oriented evaluation that decomposes utility (schema coverage, unary cell fidelity, pairwise relational consistency) and measures paper selection via a two-way QA procedure (gold→system and system→gold) with recall, precision, and F1.To support reproducible evaluation, we introduce ARXIV2TABLE, a benchmark of 1,957 tables referencing 7,158 papers, with human-verified distractors and rewritten, schema-agnostic user demands.We also develop an iterative, batch-based generation method that co-refines paper filtering and schema over multiple rounds.We validate the evaluation protocol with human audits and cross-evaluator checks.Extensive experiments show that our method consistently improves over strong baselines, while absolute scores remain modest, underscoring the task's difficulty.Our data and code is available at https://github.com