Improving Contextual Asr Via Multi-Grained Fusion With Large Language Models

Current end-to-end ASR models excel at general transcription but often struggle with context-specific keywords. Existing contextual ASR methods are typically categorized into token-level and phrase-level approaches. Token-level methods generate text step-by-step to provide fine-grained control for generating both keywords and general tokens, but often struggle to preserve the integrity of keyword phrases. In contrast, phrase-level methods model keywords as indivisible units to preserve phrase integrity, yet often degrade the recognition of non-keyword text. Prior work has not explored combining these complementary strategies. To address this gap, we propose an end-to-end multi-grained fusion framework that systematically integrates a large language model (LLM) at both granularities. For token-level fusion, it fuses the token probabilities of the ASR model and the LLM, while for phrase-level fusion, it jointly leverages the ASR model’s acoustic evidence and the LLM’s semantic guidance to select keyword phrases. A joint multi-grained mechanism then chooses the optimal output at each decoding step. Experiments on Chinese and English datasets show our approach achieves up to a 1.2% absolute error reduction and 9.8% keyword recall improvement. Ablation studies confirm both token-level and phrase-level components are essential and complementary to the final performance.