Target Speaker Isolation for Mobile Health Assessment From Speech With Crosstalk

Speech assessments are crucial for monitoring medical conditions such as Parkinson’s disease, depression, and Traumatic Brain Injury (TBI). However, crosstalk—speech from non-target speakers—presents a critical challenge in real-world mobile health assessments, often leading to false diagnoses. Continuous, real-world speech assessments using mobile devices are realistic but require additional pre-processing steps to ensure speech quality. Isolating the target speaker’s speech is necessary before health assessments. Previous research has focused on removing background noise but paid limited attention to crosstalk in passive health assessments. Some prior work addressed crosstalk using speech separation, which is found to create artifacts on the extracted speech. Instead, we propose Target Speaker Isolation with Normal Distribution (TSI-N) to isolate speech samples of the target speaker. Unlike standard embeddings, TSI-N utilizes an N-vector speaker representation trained to fit an unbounded normal distribution for each speaker cluster. This probabilistic property enables robust null-hypothesis testing and rapid online learning on resource-constrained mobile devices. In rigorous evaluations on various levels of crosstalk, TSI-N significantly outperforms state-of-the-art baselines, achieving accuracy improvements of up to 36% for TBI detection and 18% for depression detection in passive speech assessments.