Two-Stage Audio-Visual Target Speaker Extraction System for Real-Time Processing on Edge Device

Audio-Visual Target Speaker Extraction (AVTSE) aims to isolate a target speaker’s voice from multi-speaker mixtures by leveraging visual cues. However, the practical deployment of existing AVTSE methods is often hindered by poor generalization, high computational complexity, and non-causal designs. To address these issues, we propose 2S-AVTSE, a novel two-stage system built on an audiovisual decoupling strategy. This approach uniquely eliminates the need for paired audio-visual training data, enhancing its applicability in real world scenarios. The first stage uses a compact visual network to perform voice activity detection (VAD) by analyzing visual cues only. Its output VAD then guides a second-stage audio network to extract the target speech. With a computational load of only 1.89 GMACs, our system exhibits superior generalization and robustness in realistic and cross-domain scenarios compared to end-to-end baselines. This design presents a practical and effective solution for real-world applications.