Bilingual performance of ChatGPT, Gemini, and DeepSeek in asthma, allergy, and respiratory infection queries

Generative artificial intelligence (genAI) models are rapidly being adopted for health information delivery. Nevertheless, systematic cross-linguistic evaluations of their clinical reliability—particularly in high-burden conditions such as asthma, allergy, and respiratory tract infections (RTIs)—remain limited. The aim of this study was to compare the English and Arabic performance of ChatGPT‑4o, Gemini, and DeepSeek in responding to common asthma, allergy, and RTI queries using a validated clinical assessment framework. A bilingual evaluation was conducted using 30 frequently asked questions (FAQs) related to asthma, allergy, and RTIs. Each question was submitted in English and Arabic to ChatGPT‑4o, Gemini, and DeepSeek. Responses were evaluated independently by three bilingual clinical experts using the CLEAR framework for Completeness, Accuracy, and Relevance of the generated content. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Language and model comparisons were analyzed using non-parametric Kruskal-Wallis and Mann-Whitney U tests. The study followed the METRICS reporting guideline for genAI in healthcare. ChatGPT‑4o consistently outperformed Gemini and DeepSeek across all CLEAR dimensions and the two languages. In English, the mean CLEAR scores were: ChatGPT‑4o: 3.90, Gemini: 2.50, DeepSeek: 2.09. In Arabic, ChatGPT‑4o again scored highest (3.63), followed by Gemini (2.38) and DeepSeek (1.84). All inter-model differences were statistically significant (p < 0.001). Inter-rater reliability was excellent across dimensions: ICC for completeness = 0.858, accuracy = 0.917, relevance = 0.950 (all p < 0.001), confirming strong consistency and validity in scoring. Within each genAI model, English outputs significantly outperformed Arabic in completeness, accuracy, relevance, and the overall CLEAR score. Domain-wise, asthma queries achieved the highest performance across models and languages, while allergy queries showed the lowest accuracy. ChatGPT‑4o demonstrated superior bilingual performance, while Gemini and DeepSeek exhibited significant limitations, particularly in Arabic. These findings highlight persistent language-based disparities in genAI health outputs. Rigorous cross-linguistic evaluation and domain-specific fine-tuning are essential to ensure safe and equitable deployment of genAI tools in global health communication.