Label-Free Speaker Diarization Using Self-Supervised Speaker Embeddings

Neural network-based speaker embedding models have become the standard for speaker verification, identification, and diarization, offering highly discriminative speaker representations. However, strong performance typically relies on large amounts of annotated data and pretrained embedding extractors trained under conditions that may not match the target domain. Under substantial domain mismatch, directly applying pretrained representations or full diarization pipelines can degrade accuracy, motivating approaches that learn speaker-discriminative representations directly from unlabeled in-domain audio. This paper extends our previous work on entirely unsupervised deep learning for speaker diarization. Specifically, we focus on generating high-quality neural speaker representations without speaker-identity labels or diarization time annotations, and on estimating secondary hyper-parameters from unlabeled audio, using off-the-shelf VAD only. Speaker embeddings are learned in a self-supervised fashion from pairs of adjacent segments assumed to be from the same speaker. The trained encoder is then used to self-generate pseudo-labels, which are used to train a PLDA similarity scorer and to tune the clustering stopping criterion. We further improve performance by iteratively retraining the embedding model using updated pseudo-labels. We evaluate our approach on the CallHome benchmarks and include an additional experiment on air-traffic-control (ATC) communications to test a strongly domain-shifted setting. Results show that the proposed label-free pipeline outperforms prior unsupervised methods when up to three speakers are present and narrows the gap to recent supervised systems on CallHome, while providing clear gains over applying pretrained models in the ATC domain.