Clinical Med students’ validation of Arkangel AI: Are their responses any better when supported by the AI?

ABSTRACT Introduction Large Language Models (LLMs) in healthcare education have been primarily evaluated using medical question-answering datasets, often showing strong performance. However, multiple-choice formats may be insufficient to assess more complex language-based clinical reasoning. Objective To evaluate time investment and response validity among medical students answering clinical questions using Arkangel AI compared with traditional search methods. Methods A randomized, double-blind study was conducted with clinical-year medical students assigned to two groups. Participants answered four clinical questions derived from four clinical cases using either Arkangel AI or traditional search resources (e.g., Google, PubMed). Responses were independently evaluated by two specialist physicians per clinical specialty using six predefined validity criteria. Results Eighty-three medical students were randomized to Group A (Arkangel AI; n = 43) and Group B (traditional search; n = 40). Group A demonstrated a relative reduction in median time per case of approximately 55% compared with Group B (median 3:09 vs. 5:28) and required fewer searches per case (median 3 vs. 6), corresponding to an approximate 50% reduction. Responses generated using Arkangel AI showed higher validity across all predefined dimensions, with a Total Average Validity Score of 2.73 in Group A compared with 2.55 in Group B. Most participants using Arkangel AI reported high perceived usefulness and willingness to recommend the tool to peers. Conclusion LLM-supported approaches may improve the efficiency of clinical information seeking while maintaining or enhancing response validity in undergraduate medical education, particularly in non-critical clinical reasoning contexts. Further validation involving graduated physicians and specialists is warranted