Impact of an LLM-based Review Assistant in Practice: A Mixed Open-/Closed-source Case Study

Code review is a standard practice in modern software development, aiming to improve code quality.As providing constructive reviews on a submitted patch is a challenging and error-prone task, the advances of Large Language Models (LLMs) in performing natural language processing (NLP) tasks in a human-like fashion have prompted researchers to evaluate the LLMs’ ability to automate the code review process. However, outside lab settings, it is still unclear how prone reviewers are to accept comments generated by LLMs in a real development workflow. To fill this gap, we conduct a large-scale empirical case study in a live setup to evaluate the acceptance of LLM-generated comments and their impact on the review process. This case study was performed in two organizations, Mozilla (which has its codebase available as open-source) and Ubisoft (fully closedsource). Inside their usual review environment, participants were given access to RevMate, an LLM-based assistive tool that suggests generated review comments using an off-the-shelf LLM with Retrieval-Augmented Generation to provide extra code and review context, combined with LLM-as-a-Judge, to autoevaluate the generated comments and discard irrelevant cases. Based on more than 587 patch reviews, we observed that 8.1%, of LLM-generated comments were accepted by reviewers in each organization respectively, while 14.6% and 20.5% other comments were still marked as valuable as review or development tips. Refactoring-related comments are more likely to be accepted than Functional comments (18.2% and 18.6% compared to 4.8% and 5.2%). The extra time spent by reviewers to inspect generated comments or edit accepted ones (36/119), yielding an overall median of 43s per patch, is reasonable. The accepted generated comments are as likely to yield future revisions of the revised patch as human-written comments (74% vs 73% at chunk-level).