dWhisper-ASV: Distilling Knowledge Into Whisper Encoder for Fast and Domain-Robust Speaker Recognition

Recent advances in self-supervised learning have enabled highly effective automatic speaker verification (ASV) systems based on foundation models such as WavLM and wav2vec 2.0, albeit at the cost of substantial computational complexity. In this study, we address this limitation by employing the MarginDistillation (MD) framework to distill a wav2vec 2.0-based teacher model into a compact student model, dWhisper-ASV, transferring robust discriminative capabilities while achieving an eightfold reduction in computational cost. We further propose a two-stage biometric search pipeline: dWhisperASV first performs rapid candidate retrieval via cross-model comparisons with the teacher's embedding space, after which candidates exceeding a pre-filter threshold are re-ranked by the teacher model. Experiments on established ASV protocols, including recent NIST SREs, demonstrate that dWhisper-ASV outperforms comparable lightweight speaker verification systems. Evaluation of the proposed search pipeline on Speech-Massive (1.7 K speakers) and ClovaCall (7 K speakers) databases shows that the two-stage approach matches the verification quality of the teacher model while providing an eightfold acceleration in database processing.