A comparative analysis of embedded chatbot models and ChatGPT-4 for answering orthodontic treatment queries

This study evaluates the effectiveness of an embedded chatbot model in delivering orthodontic treatment-related information and compares its performance with the fourth generation of the Chat Generative Pre-trained Transformer (ChatGPT-4, OpenAI, USA), focusing on accuracy, clarity, relevance, and up-to-date knowledge. An embedded chatbot model, designed specifically for orthodontic treatment-related queries, was integrated into a web-based platform. It was compared with ChatGPT-4 (OpenAI, USA) using standardized orthodontic questions. Responses from both models were evaluated by six orthodontic consultants (≥ 5 years of experience) using a 5-point Likert scale. Content validity was assessed using item-level (I-CVI) and scale-level (S-CVI) content validity indices. Statistical analysis was performed using the Wilcoxon signed-rank test. Normality was assessed using the Shapiro-Wilk test. The embedded chatbot demonstrated higher content validity and received better scores in accuracy and clarity compared to ChatGPT-4 (OpenAI, USA). A greater number of its responses achieved acceptable I-CVI (≥ 0.78) and S-CVI (≥ 0.8) values. Although the embedded model showed numerical trends favoring its performance over ChatGPT-4 (OpenAI, USA), the differences were not statistically significant (p = 0.173). The embedded chatbot model showed improved clarity and validity in orthodontic responses, suggesting potential advantages of domain-specific embedding and prompt engineering. While performance differences were not statistically significant, the findings support the growing role of tailored artificial intelligence (AI) tools in enhancing patient communication and clinical support in orthodontics.