Benchmarking Large Language Models for Automated Code Changes: A Systematic Review

View at publisher Summarise with VIRAT
Authors: Padamjit Sandhu, Pankaj Khurana, Jimmy Singla
Cited by: 0
References: 23

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Large Language Models (LLMs) have been employed more and more to automate code changes (ACC) for a range of development projects, such as enhancements in features, refining and error remediation. Recently, a number of LLM based automate code change paradigms have been created and reviewed. Although LLMs in ACC tasks have been studied in past research, this domain still needs to evaluate uniform procedures, a standardized dataset and reducibility challenges. Therefore, this study provides a systematic analysis of existing literature on LLM based ACC tasks to examine the core data sets, frequently employed assessment metrics and persistent reducibility issues. Findings of this review study revealed that the most frequently used dataset to train an LLM model for code change is Defects4J. Similarly, repair success rate, pass@k and patch accuracy were the main evaluation measured utilized in the articles to evaluate the performance of the developed model. Moreover, this study also examined various reproducibility challenges that can be faced when recreating a similar ACC model. These results open the door to more investigation into how to fully utilized LLMs to transform automated code changes.

Fields of Study & Subjects

Software Engineering Research Software Testing and Debugging Techniques Software Engineering Techniques and Practices

Publication details

Journal / source
International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)
Pages
1459–1464
Publication date
4 February 2026
Language
EN