A Semantic Evaluation Framework for Clinical Metadata Extraction Reliability: A Pilot Study with Large Language Models
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
This paper proposes a semantic evaluation framework to assess the reliability of clinical metadata extracted from doctor-patient conversations using large language models (LLMs), addressing the limitations of traditional metrics like exact matching that fail to capture medical meaning. Developed in the context of the HealthSync system, the framework combines embedding-based semantic similarity, structured output validation, weighted precision/recall, and hallucination detection to evaluate extraction quality across multiple dimensions. Through experiments on 20 clinical conversations using four LLMs, the study demonstrates that conventional methods significantly overestimate performance, while the proposed approach effectively identifies critical issues such as semantic drift, missing information, and hallucinated medical entities. Overall, the framework provides a more realistic and clinically relevant method for evaluating AI-driven medical documentation systems, emphasizing the importance of semantic correctness and patient safety
Fields of Study & Subjects
Artificial Intelligence in Healthcare and Education
Biomedical Text Mining and Ontologies
Topic Modeling
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 28 March 2026
- Language
- EN