Continuous-Token Diffusion for Speaker-Referenced TTS in Multimodal LLMs

Unified architectures in multimodal large language models (MLLM) have shown promise in handling diverse tasks within a single framework. In the text-to-speech (TTS) task, current MLLM-based approaches rely on discrete tokens, which disregard the inherently continuous nature of speech and can lead to loss of fine acoustic detail. In this work, we investigate TTS within the MLLM paradigm using continuous speech representations. We design a dual-head architecture and implement two complementary training strategies for a robust model. (1) A diffusion head generating continuous speech representations is added to the MLLM, which is on frame-level and strictly autoregressive. (2) The original language model head is retained to preserve multitask capability and to control the start and end of speech synthesis. (3) Masked training is employed to address exposure bias in autoregressive decoding. (4) A two-stage scheme freezes the LM in stage 2, stabilizing optimization by fixing the diffusion input distribution. Evaluations on LibriSpeech (PC) test-clean show that our approach achieves state-of-the-art autoregressive performance, with a WER of 1.95%, speaker similarity of 0.54, UTMOS of 4.00, and MOS of 3.77. The two-stage training yields a 46% relative WER reduction over one-stage training. These results highlight the effectiveness of combining autoregressive modeling with continuous-token diffusion.