SLM outperform LLMs in Classifying Social Determinants of Health in Clinical Records (Preprint)

BACKGROUND Social and Behavioral Determinants of Health (SDoH) are non-clinical social and environmental factors – such as income, education, housing, and access to care – that strongly influence health outcomes, are often embedded in unstructured EHR (electronic health record) notes, and remain difficult to systematically identify despite accounting for most modifiable drivers of health disparities. OBJECTIVE This study contributed expert-derived resources to support the development of high-quality SDoH annotated datasets and compared the performance of smaller transformer-based models and large language models (LLMs) in classifying SDoH snippets. METHODS We generated high-quality SDoH snippets using keywords to extract relevant snippets from 50,000 electronic health records, categorized into 20 SDoH types with subcategories (e.g., positive, adverse, uncertain). We integrated annotation schemes from related works and applied four distinct schemes to address real-world needs. Fine-tuning was performed on smaller transformers, including BERT and Gatortron, and state-of-the-art LLMs, such as Llama-3 and Mistral. Model performance was evaluated using macro-averaged F1, Precision, and Recall scores across four annotation levels. RESULTS We curated a high-quality dataset comprising 2,364 annotated SDoH snippets. Among the four annotation levels, Gatortron-S performed best in the first two levels, BioClinical-BERT in the third level, and Llama-3 in the fourth level. Additionally, we released a comprehensive list of expert-derived keywords to facilitate future efforts in SDoH snippet annotation and dataset development. CONCLUSIONS We found that smaller transformer-based models demonstrated comparable or superior performance to advanced LLMs when sufficient data were available. This challenges the assumption that LLMs universally outperform smaller models, emphasizes the necessity of high-quality annotated datasets, and provides critical resources and insights for advancing SDoH classification research.