CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology

Abstract Context: Existing Large Language Model (LLM)-based multi-agent systems are capable of executing tasks and providing data-driven recommendations, thereby enabling automation and decision support that assist practitioners in software development. However, existing studies have tested the outcomes of agents on benchmark datasets, offering only binary pass-or-fail results, which provide limited insight into their practical applicability. There remains a lack of empirical research examining the potential and limitations of LLM-based agents in addressing challenging, real-world tasks such as automated code generation for large-scale systems. Objective: This study empirically investigates the potential of LLM-based agents in software development by engaging participants to evaluate the agents performance in autonomous software development tasks. Method: We employed a two-phase approach comprising (i) the development of a multi-agent system, CodePori, to automate code generation, and (ii) participant-based evaluation to assess agent performance and explore their practical applicability for software development. Results: The results of this study present participants’ feedback and insights into the use of LLM-based multi-agent systems in real-world software development, including their strengths, challenges, and areas for improvement. We also highlight key aspects missed by code-generation benchmarks, which collectively enhance our understanding of the practical applicability of LLM-based agents. Conclusions: Based on the results, we conclude that, while LLM-based multi-agent systems show potential for large-scale software development, their successful integration requires the addressing of specific challenges (e.g., shortterm memory limitations, hallucinations, and code smells) and incorporating a practitioner-centric perspective. The study highlights the need to move beyond standard benchmarks to evaluate real-world applicability and identifies new opportunities for broader adoption in both industry and academia.