Proficiency-Aware Adaptation and Data Augmentation for Robust L2 ASR

General-purpose automatic speech recognition (ASR) underperforms for atypical speakers, such as second-language (L2) learners, reinforcing bias and limiting use in education and accessibility. Using the Common European Framework of Reference for Languages (CEFR)-graded Speak & Improve corpus, we show that naïve fine-tuning of Whisper reduces the average word error rate (WER) but simultaneously widens performance disparities and disproportionately harms lower-proficiency learners. To address this, we propose two strategies: (i) proficiency-aware multitask learning, jointly optimizing ASR with proficiency classification, and (ii) targeted augmentation, applying spectrogram masking to low-proficiency speech to counter imbalance. These approaches reduce WER by up to 29.4% (relative) and insertion/deletion errors by as much as 58.6% (relative). Crucially, despite the dataset’s severe imbalance reflecting real-world distributions, both strategies consistently narrow proficiency gaps, advancing equitable ASR for L2 learners.