A deep neural network-based automatic mispronunciation detection in Bengali accented English speech

Abstract Learning a second language, especially English, became necessary as globalisation started. One crucial component of language learning resources is computer-assisted pronunciation training, or CAPT. Automatic mispronunciation detection (AMD) is the central feature of CAPT. With AMD, errors in pronunciation are automatically diagnosed, which is important for students to recognise their mispronunciation and work on their articulation. However, in non-native English speech scenarios, most AMDs fail to identify errors and give feedback. Moreover, AMD research on Bengali English learners faced many problems due to a lack of relevant data, including this, in the low-resource scenario. In this paper, we create a corpus of English speech with Bengali accents, focusing on the research of non-native AMD tasks. A framework for AMD has also been proposed, which employs a branchformer to capture local and global speech context and end-to-end attention connectionist temporal classification (CTC) models to identify pronunciation errors. We conducted our experiment with our newly developed speech dataset. Experimental results show that our branchformer encoder-based attention-CTC AMD model outperforms the baseline convolutional neural network-recurrent neural network-connectionist temporal classification (CNN-RNN-CTC) based architecture by 6.89% based on the F1-score. Mispronunciation detection accuracy is increased to 90.08% from 74.82%. The phoneme error rate is also improved from 32.07% to 10.80% on our created dataset compared to the baseline model.