Development and evaluation of machine learning models for the detection of emergency department patients with opioid misuse from clinical notes

Objectives: The accurate identification of Emergency Department (ED) encounters involving opioid misuse is critical for health services, research, and surveillance. We sought to develop natural language processing (NLP)-based models for the detection of ED encounters involving opioid misuse. Methods: A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was benchmarked to opioid-related ICD-10-CM codes. Both raw text and text transformed to the Unified Medical Language System were evaluated. Face validity was evaluated by term feature importance. Results: There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC 0.9358 (95% CI 0.8945, 0.9681), accuracy 0.8874 (0.8402, 0.9349), and F1 score 0.8624 (0.7969, 0.9197) which performed comparably to ICD-10-CM codes [accuracy 0.8687 (0.8155, 0.9167); F1 0.8296 (0.7544, 0.8939)]. Excluding XGBoost, fine-tuned pre-trained language models generally outperformed classic ML methods. The best performing model by point estimate was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.9474 (0.9113, 0.9749); accuracy 0.8816 (0.8284, 0.9290); F1 0.8499 (0.7805, 0.9103)] but confidence intervals overlapped with other models. Explainability analyses showed the most predictive terms were "heroin," "opioids," "alcoholic intoxication, chronic," "cocaine," "opiates," and "suboxone." Conclusions: NLP-based models perform comparably to entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.