Multi-Modal Feature Fusion and Stacking Ensemble Learning for Lyric Intelligibility Prediction

Predicting lyric intelligibility across varying levels of hearing loss is challenging. This study investigates a multi-modal feature set—including acoustics, speech recognition error metrics, and linguistic features—for the prediction. The proposed method achieved significant improvements over the baseline Whisper model on the Cadenza dataset. Specifically, the root mean square error on the evaluation set was reduced from 29.08% to 27.22%. This study identifies specific acoustic and linguistic features, such as prosodic cues and text embeddings, that drive the differences in how speech and lyric intelligibility are perceived.