Improving drug identification in overdose death surveillance by using clinical natural language processing models

Abstract The rising rate of drug‐related deaths in the United States, largely driven by fentanyl, requires timely and accurate surveillance. However, critical overdose data are often buried in free‐text coroner reports, leading to delays and information loss when coded into ICD (International Classification of Disease)‐10 classifications. Natural language processing (NLP) models may automate and enhance overdose surveillance, but prior applications have been limited. A dataset of 35,433 death records from multiple US jurisdictions in 2020 was used for model training and internal testing. External validation was conducted using a novel separate dataset of 3335 records from 2023 to 2024. Multiple NLP approaches were evaluated for classifying specific drug involvement from unstructured death certificate text. These included traditional single‐ and multi‐label classifiers, as well as fine‐tuned encoder‐only language models such as Bidirectional Encoder Representations from Transformers (BERT) and BioClinicalBERT, and contemporary decoder‐only large language models (LLMs) such as Qwen 3 and Llama 3. Model performance was assessed using macro‐averaged F 1 scores, and 95% confidence intervals were calculated to quantify uncertainty. Fine‐tuned BioClinicalBERT models achieved near‐perfect performance, with macro F 1 scores ≥0.998 on the internal test set. External validation confirmed robustness (macro F 1 = 0.966), outperforming conventional machine learning, general‐domain BERT models, and various decoder‐only LLMs. NLP models, particularly fine‐tuned clinical variants like BioClinicalBERT, offer a highly accurate and scalable solution for overdose death classification from free‐text reports. These methods can significantly accelerate surveillance workflows, overcoming the limitations of manual ICD‐10 coding and supporting near real‐time detection of emerging substance use trends.