Assessing the clinical competence of large language models for tobacco use disorder: A multi-domain expert evaluation

BACKGROUND: Tobacco use disorder (TUD) remains the leading preventable cause of death globally, yet fewer than one-third of users receive guideline-concordant care due to workforce shortages and training gaps. Emerging artificial intelligence (AI) systems, particularly large language models (LLMs), may help expand access to evidence-based cessation support, but their clinical competence remains insufficiently characterized. OBJECTIVE: To systematically evaluate the clinical accuracy, safety, guideline adherence, and communication quality of five leading LLMs across standardized tobacco-cessation scenarios. METHODS: We developed 84 clinical vignettes, covering five core competency domains (screening, diagnosis, pharmacotherapy, behavioral counseling, and harm reduction) based on DSM-5-TR, U.S. Public Health Service Guidelines, NICE NG209, ASAM guidelines, SAMHSA protocols, and the WHO MPOWER framework. Five LLMs, GPT-4.5, Claude 3.5 Sonnet, Gemini 2.5 Pro, Llama 3.1-70B, and DeepSeek-V3, were independently evaluated by four addiction-medicine experts across clinical accuracy, guideline adherence, safety, and clinical utility. RESULTS: = 3.72; 52% ≥4.0). Open-weight models (Llama 3.1-70B, M = 3.48; DeepSeek-V3, M = 3.35) lagged behind overall, with only 34-38% achieving the ≥4.0 benchmark. CONCLUSION: All evaluated AI systems demonstrated competence in tobacco-cessation counseling, but GPT-4.5 and Claude 3.5 Sonnet reached performance levels consistent with supervised clinical use and safety-critical scenarios. Clinician oversight remains essential for all medication-based interventions, and open-weight models warrant further validation before consideration for clinical implementation.