Optimizing LLM Chatbot to Human Handover with Multi-Agent AI Approach

Chatbots have traditionally streamlined the customer journey by automating routine inquiries and identifying precise moments for human intervention. Although Large Language Models (LLMs) and agentic workflows present a transformative opportunity for customer service, advancement in this domain is currently constrained by the scarcity of specialized public datasets required to evaluate and benchmark robust LLMto-human handover protocols. To address this gap, we introduce the QiscusCS dataset as an evaluation benchmark, a novel corpus of over 4,000 utterances from 100 customer service dialogues, manually annotated by domain experts. Using this new resource, this paper proposes and compares two distinct architectures for managing the handover: a streamlined Single-Agent model and a collaborative Multi-Agent system. Our experiments show that the Multi-Agent architectures achieve a Macro F1-Score of 0.6260 for Gemini 2.5 Flash and 0.6159 for GPT 4.1. On the other hand, the Single-Agent architectures show inferior performance by achieving a Macro F1-Score of 0.3333 for Gemini 2.5 Flash and 0.5356 for GPT 4.1. Our results demonstrate the viability of multi-agent architectures for enhancing chatbot efficacy. This research advances the field by benchmarking various agentic workflows and clarifying the interplay between model size and architectural strategy in optimizing customer service interactions.