Benchmarking Large Language Models for Automated Code Changes: A Systematic Review
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Large Language Models (LLMs) have been employed more and more to automate code changes (ACC) for a range of development projects, such as enhancements in features, refining and error remediation. Recently, a number of LLM based automate code change paradigms have been created and reviewed. Although LLMs in ACC tasks have been studied in past research, this domain still needs to evaluate uniform procedures, a standardized dataset and reducibility challenges. Therefore, this study provides a systematic analysis of existing literature on LLM based ACC tasks to examine the core data sets, frequently employed assessment metrics and persistent reducibility issues. Findings of this review study revealed that the most frequently used dataset to train an LLM model for code change is Defects4J. Similarly, repair success rate, pass@k and patch accuracy were the main evaluation measured utilized in the articles to evaluate the performance of the developed model. Moreover, this study also examined various reproducibility challenges that can be faced when recreating a similar ACC model. These results open the door to more investigation into how to fully utilized LLMs to transform automated code changes.
Fields of Study & Subjects
Software Engineering Research
Software Testing and Debugging Techniques
Software Engineering Techniques and Practices
Publication details
- Journal / source
- International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)
- Pages
- 1459–1464
- Publication date
- 4 February 2026
- Language
- EN