An LLM-Based Retrieval-and-Reasoning Pipeline to Extract Cannabis Use Status From EHR Clinical Notes: Algorithm Development and Benchmarking Study (Preprint)

BACKGROUND Cannabis use in the United States is rising and now exceeds daily alcohol use, particularly in young adults, yet use status is poorly captured in structured electronic health record (EHR) data, where diagnosis codes identify only a minority of users. The richest signal lies in free-text notes, where mentions are sparse and heterogeneous and entangle negation, temporality, and subject. Prior cannabis natural language processing (NLP) is limited, with no systematic model comparison, and privacy and cost make brute-force large language model (LLM) inference impractical. OBJECTIVE To develop and evaluate a locally deployable pipeline that extracts cannabis-use status from clinical notes, and to quantify how much model scale the task requires. METHODS Using a retrospective corpus of 1,000 patients (132,500 notes, ~71.5 million tokens) from all care settings of one academic health system, within its secure environment, we built a 2-stage pipeline. Stage 1 performs training-free, high-recall retrieval over the full corpus, pairing a deterministic boundary-aware lexical matcher over a compacted cannabis lexicon with zero-shot biomedical named-entity recognition. Stage 2 resolves each snippet through a clinical decision tree of 4 binary nodes (genuine mention, confirmed patient-specific status, polarity, and temporality) terminating in 5 mutually exclusive status classes, each node a separately fine-tuned open-weight language model. We benchmarked 12 open-weight models spanning 270M to 120B parameters under an identical fine-tuning and evaluation protocol with 10-fold patient-level cross-validation (primary metric, macro-averaged F1). We validated retrieval on a 100-note gold standard and reasoning on 6,146 annotated snippets (κ = 0.92 and 0.99), and compared patient-level yield against diagnosis codes. RESULTS Retrieval recovered 99.4% of gold mentions while confining language-model inference to 0.27% of corpus tokens. The strongest model (GPT-OSS-120B) reached a macro-F1 of 0.937 and was statistically superior to the other 11, though their scores clustered tightly (0.905 to 0.924). Node-level analysis traced nearly all differences to one decision, temporal grounding (current or past use; across-model range 0.055, versus ≤0.015 for other nodes). The decision-tree decomposition outperformed a flat 5-class classifier by 0.045 macro-F1, exceeding the 0.032 from the full 444-fold size range, and biomedical pretraining conferred no advantage (MedGemma-27B did not exceed its base Gemma-3-27B; P = .44). At the patient level, the pipeline identified 183 cannabis users, more than 3 times the 52 identified via diagnosis codes, plus 101 confirmed non-users, which codes cannot represent. CONCLUSIONS A retrieval-and-reasoning pipeline extracts cannabis-use status accurately while confining inference, and the protected text it touches, to under 1% of the corpus. Small open-weight models fine-tuned per node suffice; added scale helps essentially only with temporal grounding. The auditable decomposition improves accuracy, localizes the remaining bottleneck, and yields a more complete cohort than diagnosis codes. Because every component runs locally on open weights, the pipeline is deployable at population scale.