Pseudo-Labeling Based Unsupervised Domain Adaptation for LLM-Based ASR

Large Language Model (LLM) has been gradually adopted in Automatic Speech Recognition (ASR) task due to its strong long-context modeling capability, achieving remarkable performance. However, similar to other end-to-end ASR models, LLM-Based ASR (LLM-ASR) models encounter performance degradation when applied across domains. To address this issue, we propose LLM-Based Pseudo-Labeling (LPL), which leverages unlabeled audio data in the target domain to train a domain-adapted LLM-ASR model. Specifically, we propose Hybrid Pseudo-Label Decoding (HPLD) method to refine the quality of pseudo-labels in a quick non-autoregressive manner, achieving a balance between training performance and computational efficiency. Additionally, we introduce the CTC Assisted Prompt (CAP) method, which offers additional semantic hints to enhance the accuracy of transcriptions generated by the LLM. During training, we incorporate CTC loss at the post-projector position to align speech embedding prompt more effectively and feed masked CTC hypothesis embedding into the LLM prompt for further improvement. Experimental results demonstrate that LPL achieves state-of-the-art performance compared to other mainstream pseudo-labeling methods, further enhancing the cross-domain performance of LLM-ASR models.