Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
Abstract
While large language models (LLMs) have shown efficacy in automated code review, existing benchmarks suffer from three limitations: (1) a lack of rich semantic context like issue descriptions; (2) data quality issues due to insufficient validation; and (3) coarse granularity (file or commit level) that misses fine-grained, line-level nuances. To address these, we present ContextCRBench, a high-quality, context-rich benchmark designed for fine-grained evaluation of LLMs in code review tasks.
Fields of Study & Subjects
Software Engineering Research
Software Testing and Debugging Techniques
Software Engineering Techniques and Practices
Publication details
- Journal / source
- Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering
- Publisher
- ACM
- Pages
- 620–631
- Publication date
- 5 July 2026