A Semantic Evaluation Framework for Clinical Metadata Extraction Reliability: A Pilot Study with Large Language Models

Read full text Summarise with VIRAT
Authors: Sashank Santosh Kumar Mishra, Suyash Manohar Baviskar, Suvarna Pansambal, Tejinder Singh Hunjan
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

This paper proposes a semantic evaluation framework to assess the reliability of clinical metadata extracted from doctor-patient conversations using large language models (LLMs), addressing the limitations of traditional metrics like exact matching that fail to capture medical meaning. Developed in the context of the HealthSync system, the framework combines embedding-based semantic similarity, structured output validation, weighted precision/recall, and hallucination detection to evaluate extraction quality across multiple dimensions. Through experiments on 20 clinical conversations using four LLMs, the study demonstrates that conventional methods significantly overestimate performance, while the proposed approach effectively identifies critical issues such as semantic drift, missing information, and hallucinated medical entities. Overall, the framework provides a more realistic and clinically relevant method for evaluating AI-driven medical documentation systems, emphasizing the importance of semantic correctness and patient safety

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Biomedical Text Mining and Ontologies Topic Modeling

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
28 March 2026
Language
EN