Comment on ‘Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions’

I read with great interest the research report by Gorst et al. demonstrating that an expert-trained generative artificial intelligence (AI), ‘AI-Brendan’, outperformed both ChatGPT-4.0 and human experts in answering tracheostomy care questions [1]. The integration of domain-optimised AI into clinical education represents a highly promising frontier for specialities like otorhinolaryngology and critical care. However, I wish to highlight a methodological consideration regarding the evaluation of these AI-generated responses. The authors utilised a modified binary Ensuring Quality Information for Patients (EQuIP) tool, adjudicated by non-specialist university students. While this demonstrates the software's ability to produce accessible content for lay audiences, it may fail to capture subtle clinical details or experience hallucinations. Large language models, despite fine-tuning, remain susceptible to generating plausible but clinically flawed outputs. As highlighted in recent literature, robust evaluation of medical AI necessitates rigorous appraisal by domain experts to ensure patient safety and clinical fidelity [2]. An EQuIP score evaluated by non-clinicians might overvalue structural coherence and length while missing nuanced omissions which an experienced clinician would identify; for example, AI-Brendan produced significantly longer responses. Furthermore, moving from isolated question prompts to continuous conversational testing in complex clinical scenarios is essential [3]. I commend the authors on their innovative work, but advocate that future validation of technological solutions in airway management must incorporate expert clinical adjudicators to truly establish the safety and accuracy of AI instructors.