A MODULAR BENCHMARKING FRAMEWORK FOR EVALUATING LLM-BASED AGENT APPLICATIONS

Large Language Model (LLM) based agent applications integrate reasoning, retrieval, tool invocation, and multi-turn interaction to solve complex tasks.While recent advances have enabled rapid adoption of agentic systems in production, evaluation methodologies have largely remained model-centric, focusing on single-turn accuracy or preference-based scoring.These approaches fail to capture agent-level behavior such as contextual grounding, conversational fidelity, tool effectiveness, and safety characteristics.This paper introduces a modular benchmarking framework for evaluating LLMbased agent applications across models, agent architectures, and execution strategies.The framework decouples datasets, agents, models, and metrics, enabling systematic and reproducible evaluation.We define a multi-dimensional metric taxonomy spanning output quality, contextual grounding, conversational relevance, and safety, and combine it with trace-aware execution to support fine-grained analysis of agent behavior.Using representative synthetic benchmarks derived from real-world agent workloads, we demonstrate that agent design choices-particularly retrieval Karthik Perikala