Truth Is Better Generated than Annotated: Hierarchical Prompt Engineering and Adaptive Evaluation for Reliable Synthetic Knowledge Dialogues

Large Language Models (LLMs) have demonstrated exceptional performance in knowledge-based dialogue generation and text evaluation. The importance of synthetic data is also highlighted in terms of generating high-quality but low-cost datasets. However, LLM-generated synthetic data often suffers from hallucinations, inconsistencies, and self-anthropomorphized responses. Additionally, constructing knowledge-based dialogue datasets manually presents limitations, including high costs and inherent subjectivity.\\To address these challenges, we propose a hierarchical prompt engineering approach that mitigates hallucinations and self-anthropomorphized utterances while maintaining response consistency through a hybrid method, \textbf{A}utomatic \textbf{C}onstruction of Knowledge-Grounded and \textbf{E}ngaging Human-AI Conversation Dataset (ACE). Furthermore, existing evaluation methods—both human and automated—struggle to assess critical aspects of knowledge-based dialogue datasets, such as factual accuracy, coherence, and self-anthropomorphizing. To overcome these limitations, we introduce the \textbf{T}ruthful \textbf{A}nswer \textbf{S}core (TAS), a novel evaluation metric designed specifically for knowledge-based dialogue evaluation.\\Our experimental results demonstrate that our proposed dataset achieves higher quality compared to existing datasets, such as Wizard of Wikipedia (WoW) and FaithDial. Additionally, TAS aligns more closely with human judgment, providing a more reliable and scalable evaluation framework. Our findings validate the effectiveness of LLM-based synthetic data construction and assessment, reducing the dependency on human annotations while improving dataset quality.