Guided Clinical Reasoning with LLMs: Reconstructing the PHQ-9 from Multimodal Inputs

Objective: This study evaluates whether general-purpose large language models (LLMs) can detect depressive symptoms from multimodal data—physiological signals, emotional self-reports, and narrative transcripts—by performing zero-shot, symptom-level inference aligned with the PHQ-9 scale.Methods: Using the MERSA dataset, which includes Fitbit metrics (e.g., heart rate, sleep, circadian rhythm), daily PANAS surveys, and short self-narratives, we prompted LLMs (GPT-4o, GPT-4o mini, DeepSeek-R1) to infer PHQ-9 symptoms. We tested different prompting strategies and conducted ablation trials to evaluate the independent and combined contributions of each modality.Results: GPT-4o mini and DeepSeek-R1 achieved strong performance (mean F1 scores of 0.761 and 0.764), particularly for somatic symptoms like fatigue and sleep disturbance (F1 ≈ 0.9). Suicidal ideation and appetite change were not reliably predicted. GPT-4o mini favored precision; DeepSeek-R1 favored recall. Models could infer psychological symptoms from physiological data and vice versa, suggesting mind–body reasoning. Input pipeline variations had minimal impact. Transcripts added little value due to low emotional content.Conclusion: General-purpose LLMs can reason over structured and heterogeneous health data without task-specific fine-tuning. Their capacity to infer depressive symptoms across mental and physical domains and to tolerate missing or uninformative modalities supports their use in modular mental health systems. Differences in model behavior suggest that LLM selection should be guided by desired recall-precision trade-offs in clinical applications.