Sanora: A Conversational AI Agent for Multimodal Digital Biomarkers of Mental Health

Multimodal data can yield digital biomarkers relevant to depression and overall mental health. This study aimed to (1) extract an evidence-based library of vision, speech, and language biomarkers; (2) assess the feasibility of a fully remote conversational platform (Okaya) for collecting multimodal data; and (3) conduct preliminary signal checks for depression, fatigue, and cognition. Participants (N=10) were recruited from a women's mental health group in Australia and completed a total of 59 sessions. During the study, participants completed a check-in via \textit{Sanora}, a conversational AI agent with an integration to a large-language model (LLM). From these interactions, 66 visual, acoustic, and text features were extracted. Validated assessments were also collected: PHQ-9 (depression), Cancer Fatigue Scale (fatigue), and Trail Making Test (cognition). We explored correlations between extracted features and assessments. Preliminary correlations identified promising digital biomarkers for depression (average F5 formant frequency and sentiment score), fatigue (segmented TTR text complexity and sentiment score) and cognition (volume, harmonicity, spectral entropy, pause length standard deviation, and eyelid droop). We demonstrate feasibility of the conversational AI-enabled platform for extracting digital biomarkers in a sample of adults with depression. Taken together, our findings align with the use of previously discovered digital biomarkers as a preliminary signal check, inform the development of personalized, remote monitoring models for mental health, and generate hypotheses for larger pilot and validation studies.