Cross-Modal Diffusion on Pretrained Alignment Codebook for Multimodal Machine Translation

Multimodal machine translation aims to integrate auxiliary information, such as images, to enhance the quality of text translation. While existing methods deliver promising performance through various cross-modal fusion techniques, they all ignore the urgent need for decoding speed during inference, which limits their applicability in real-world scenarios. In this paper, we propose a novel multimodal diffusion translation framework calledMDT, which balances performance and efficiency to generate high-quality target translation in parallel. Specifically, we design an alignment codebook with shared semantics, which facilitates efficient correlation between image and text inputs through latent alignment and multimodal alignment pretraining tasks. A cross-modal diffusion model is built on the codebook to recover latent text codewords efficiently using visual features. We also develop a Bézier noise schedule, which allows the denoising difficulty measured by KL-divergence to grow robustly with the time step. We also develop a Bézier noise schedule, which allows the denoising difficulty measured by KL-divergence to grow robustly with the time step. Experimental results on two datasets with five bilingual language pairs demonstrate the effectiveness ofMDTover advanced methods, achieving superior inference speedup.