Text mining methods for automated data extraction from health technology assessment reports of medicines using classical natural language processing and generative artificial intelligence

Objective: This proof of concept for utilizing automatic data extraction methods to extract health technology assessment (HTA) attributes from HTA reports of medicines aimed to explore which attributes could be extracted and how accurately, using different data extraction methods. This enables easy access to insights into HTA recommendations for policymaking and policy-related research. Materials and Methods: In total, 14 relevant attributes (eg, assessment outcome or date) were identified for extraction using two classical natural language processing (NLP) methods (rule-based and classification models) and a generative AI method (large language model (LLM)-based, i.e., Claude 3 Opus). The performance of these techniques was compared using 50 HTA reports published by the National Institute for Health and Care Excellence (NICE, United Kingdom). Results: All three methods were able to extract certain attributes with high accuracy, with differences between the extraction methods and the type of attribute. The LLM-based extraction was the only method able to extract attributes on a medicine-indication combination level. The LLM-based extraction performed best (88-98% semantical accuracy for 12/14 attributes). Extraction of Outcome relative effectiveness analyses (REA) and Comparator was the most challenging and had the lowest accuracy (∼70% for the LLM-based extraction). Discussion & Conclusion: Automatic data extraction for relevant attributes from HTA reports is possible, but there is still room for improvement. LLM-based extraction outperformed the two NLP methods, but challenges regarding the use of commercial software and reproducibility remain. Future research should focus on expanding the system to other HTA organizations and further refining the LLM-based extraction.