ChatGPT Versus Custom-Trained Chatbot for Urogynecology Surgery Counseling

IMPORTANCE: Patient education materials are important components of shared decision making in urogynecologic surgery. Traditional materials are often difficult to update, lack personalization, and are not easily adaptable. Chatbots offer a new approach to generating comprehensive and accessible content, but their utility in a clinic setting remains unclear. OBJECTIVE: The objective of this study was to compare the performance of a general-purpose chatbot, ChatGPT, and a domain-specific chatbot developed by the Foundation for Female Health Awareness (FFHA) (FFHA Assistant) in generating surgical counseling information, using standardized materials from the International Urogynecological Association (IUGA) as reference. STUDY DESIGN: Seven IUGA handouts representing common urogynecologic surgical procedures were selected. Identical prompts were submitted to ChatGPT-4.0 and the FFHA Assistant. Responses were reviewed by 7 blinded urogynecology experts using 5-point Likert scales to assess accuracy, completeness, and understandability. Readability was evaluated using the Flesch-Kincaid Grade Level and Flesch Reading Ease Score. RESULTS: ChatGPT-4.0 outperformed in completeness as compared with the IUGA leaflets (median 4 [3-5] vs 3 [3-4], P