Evaluating reasoning large language models with human-like thinking in ophthalmic question answering

OBJECTIVES: To evaluate the performance of reasoning large language models (LLMs) with human-like thinking in ophthalmic question answering. METHODS: We evaluated two state-of-the-art open-source reasoning LLMs (DeepSeek-R1 and QwQ-32B) and one conventional non-reasoning LLM (LLaMA-3.3-70B-Instruct) models on ophthalmology questions, assessing not only answer accuracy (ACC) but also the quality of their reasoning processes. First, we curated MedQA-Eye, a dataset of 967 ophthalmology questions across 10 subspecialties, 3 scenarios, 5 medical entities and 3 languages. Second, we proposed a novel framework considering human thinking patterns essential to medical practice to evaluate the thinking performance of reasoning LLMs on MedQA-Eye. RESULTS: DeepSeek-R1 demonstrated superior overall ACC (90.59%, 95% CI 88.59% to 92.27%) to LLaMA-3.3-70B-Instruct (87.90%, 95% CI 85.69% to 89.81%, p=0.015) and QwQ-32B (84.28%, 95% CI 81.85% to 86.44%, p