Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models

Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding substantially reduces the performance gap between monolingual and bilingual models on zero-shot phonetic discrimination, achieving a 74.7% relative reduction. Indeed, while both monolingual and bilingual models benefit from visual grounding, this improvement is particularly pronounced for bilingual models, which benefit 20.05% relatively more than their monolingual counterparts.