Speaker-Conditioned U-Shaped Diarization With Speaker Extraction-Guided Enhancement

Speaker diarization demarcates speech segments by speaker, answering the question “who spoke when?”. Recently, a promising approach has emerged by integrating speaker diarization with speech separation or speaker extraction, which offers better generalization and requires significantly less pretraining data. Despite this progress, efficiently aligning speaker extraction with the inherent requirements of diarization remains a key challenge. In this paper, we introduce Speaker-conditioned U-shaped Diarization with Speaker Extraction-guided Enhancement (SUDx), a joint network where speaker diarization is the primary target and speaker extraction serves as a supportive auxiliary task. SUDx leverages the U-net architecture and exploits hierarchical speaker representation to enhance the performance of speaker diarization. In addition, SUDx does not rely on explicit speaker identity labels for supervision, allowing it to learn distinctive acoustic characteristics and adapt easily to real-recorded multi-speaker conversations. Furthermore, we introduce a novel inference strategy that effectively handles unknown number of speakers and reduces reliance on large-scale pretraining data. We show that SUDx outperforms competitive baselines for speaker diarization while maintaining high-quality speech extraction on the LibriMix dataset. We further assess our proposed approach and our novel strategy on the AMI and AISHELL-4 meeting corpora, experimental results indicate that our model achieved state-of-the-art performance with much less pretraining data.