Leveraging natural language processing and machine learning to identify chronic conditions from primary care electronic medical records

Primary care electronic medical records (EMRs) contain rich data that can support proactive identification of chronic health conditions. However, leveraging unstructured EMR data requires the use of novel computational methods. We applied natural language processing and machine learning (ML) techniques to structured and unstructured EMR data to detect arthritis, chronic kidney disease, diabetes, hypertension, and respiratory diseases. Using data from 449 community-dwelling older adults in one Canadian primary care clinic, we developed an analytical pipeline that included preprocessing of unstructured data, Latent Dirichlet Allocation topic modelling, and supervised ML models (regularized logistic regression [RLR], support vector machine [SVM], artificial neural networks [ANNs]) with class-weighted learning and Synthetic Minority Oversampling Technique techniques to address class imbalance. Integrating unstructured clinical notes improved model performance, particularly for conditions often under-coded in structured data. For example, the area under the receiver operating characteristic curve increased from 0.724 to 0.841 for SVM classifiers in arthritis detection and from 0.733 to 0.890 for ANNs in respiratory disease detection. Less pronounced improvements were observed for diabetes, hypertension, and CKD. These findings highlight that while performance gains from unstructured data vary by condition, leveraging these data can improve disease detection in primary care EMR data.