Bugs in Modern LLM Agent Frameworks: An Empirical Study

Read full text
Authors: Xinxue Zhu, Jiacong Wu, Xiaoyu Zhang, Tianlin Li, Yanzhou Mu, Juan Zhai, Chao Qun Shen, Chunrong Fang, Yang Liu
Cited by: 1
References: 2

Abstract

LLM agents have been widely adopted in real-world applications, relying on agent frameworks for lifecycle execution and multiagent coordination. As these systems scale, understanding bugs in the underlying agent frameworks becomes critical. However, existing work mainly focuses on agent-level failures, overlooking framework-level bugs. To address this gap, we conduct an empirical study of 998 issue reports from CrewAI and LangGraph, and construct a taxonomy of 15 root causes and seven observable symptoms across five agent lifecycle stages: 'Agent Initialization', 'Perception', 'Self-Action', 'Mutual Interaction' and 'Evolution'. Our findings show that agent framework bugs mainly arise from 'API Misuse', 'API Incompatibility', and 'Documentation Desync', largely concentrated in the 'Self-Action' stage. Symptoms typically appear as 'Functional Error', 'Crash', and 'Build Failure', reflecting disruptions to task progression and control flow.

Publication details

Journal / source
Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering
Publisher
ACM
Pages
1588–1592
Publication date
5 July 2026