AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation

We present AudioGen-Omni, a unified approach based on multimodal diffusion transformers (MM-DiT) that generates high-fidelity audio, speech, and songs with coherent video synchronization. Central to our approach is a lyrics-transcription encoder that maps graphemes and phonemes from speech and song into dense frame-level representations, serving as a unified semantic-acoustic bridge across modalities. To achieve precise cross-modal synchronization, we introduce phase-aligned anisotropic positional infusion(PAAPI), which selectively applies Enhanced Transformer with Rotary Position Embedding(RoPE) within an AdaLN-based joint attention module to enforce fine-grained cross-modal alignment, enhancing lip-sync accuracy and temporal coherence. Unlike text-frozen paradigms, AudioGen-Omni unfreezes all modalities and adopts masked input strategies to handle missing data, enabling flexible cross-modal conditioning. Jointly trained on large-scale video-text-audio corpora, it achieves superior semantic alignment, audio quality, and adaptability across text-to-audio/speech/song generation task, while generating 8 seconds of audio in just 1.91 seconds. Demo samples are available at: https://ciyou2.github.io/AudioGen-Omni/.