Artifacts for "Planner-Coder Gap Study"

Read full text Summarise with VIRAT
Authors: anonymous authors
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Artifact: source code and data Although the multi-agent systems (MASs) have demonstrated impressive codegeneration performance, their robustness remains poorly understood, raisingconcerns for their deployment to handle various user inputs. Thispaper conducts a systematic study to uncover the internal robustness flaws ofMASs across multiple backend LLMs and datasets using a testing pipeline thatincorporates semantic-preserving mutation operators and a novel fitnessfunction. Our findings reveal substantial robustness flaws of MASs: semanticallyequivalent inputs cause drastic performance drops, with MASs failing to solve7.9\%--83.3\% of the mutated problems they initially resolved successfully beforemutation. Our further failure analysis uncovers a fundamental cause underlying theserobustness issues: the planner-coder gap, which accounts for 75.3\% of failures.This gap arises from information loss in the multi-stage transformation process,where planning agents decompose requirements into underspecified plans whilecoding agents subsequently misinterpret intricate logic during code generation.Based on this formulated information transformation process, we propose arepairing method to mitigate information loss through multi-prompt generationand introduce a monitor agent to bridge the planner-coder gap. Evaluation showsthat our repairing method effectively enhances the robustness of MASs by solving40.0\%--88.9\% of identified failures and avoiding up to 85.7\% of failures whentesting on the repaired MASs. Our work uncovers critical robustness flaws inMASs and provides effective mitigation strategies, contributing essentialinsights for developing more reliable MASs for code generation. Our artifact include more experiment result for RQ2 and RQ3, prompt and examples for the mutation operators and monitor agent, and source code and experiment result of our paper. Prerequisites For each MAS (Self-Collaboration, MetaGPT and PairCoder), you can find ` requirement.txt` in corresponding folders. run` pip install -r requirement.txt ` to prepare the environment for each MAS. Source code and experiments Source code and experiment result for Self-Collaboration Code Generation (SCCG) can be found in `./Experiments/Self-Collaboration`. You can and run the scripts under `./sh` to regenerate the result. Source code and experiment result for MetaGPT can be found in `./Experiments/metagpt`. You can and run the scripts under `./z_scripts` to regenerate the result. Source code and experiment result for PairCoder can be found in `./Experiments/PairCoder`.You can and run the scripts under `./z_scripts` to regenerate the result. Full Experiment Result Since the raw output exceed the limit of Zenodo, we leave the full experiment result in https://github.com/anonymous-for-submit/MAS_testing_repairing. More Experiment Result The full result of RQ2 can be found in `./More_results/RQ2_all_result.pdf` Prompt and examples for the mutation operators and monitor Prompt and examples for the mutation operators can be found in `./More_results/Mutation_opertors/operators.pptx` Prompt and examples for the monitor agent can be found in `./More_results/Monitor/monitor.pptx` Annotation result for MAS failure reasons Manual annotation for MAS failure reasons are available in `./More_results/failure_categorization.xlsx`

Fields of Study & Subjects

Software Engineering Research Software Testing and Debugging Techniques Evolutionary Algorithms and Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
30 March 2026
Language
EN