Mind the gap: evaluating domain-specific language models beyond general benchmarks in exercise health AI

Standard benchmarks used for general large language models (LLMs) are inadequate for evaluating specialized models in the high-stakes exercise health domain, where safety, personalization, and deep domain knowledge are paramount. Current evaluation practices, including expert scoring, knowledge-based Q&A, and user feedback, offer limited insights into real-world applicability and clinical robustness. This discussion highlights these critical evaluation gaps and argues for the necessity of a multi-faceted, domain-specific evaluation framework. We propose incorporating strategies such as contextualized scenario simulations, comparative benchmarking against professional tools, structured cross-disciplinary audits, iterative real-world feedback loops, and rigorous adversarial safety testing. Developing and adopting such tailored, comprehensive evaluation methods is crucial for ensuring the reliability, safety, and effectiveness of LLMs, thereby fostering trust and enabling their responsible integration into exercise health practice to benefit athletes and patients.