Hard-coded Modality Routing in Shared-Attention Transformers: A Toy-Scale Empirical Study

Training multiple modalities simultaneously without inter-modality interference is a key challenge in multimodal transformers. We propose a Shared Attention + Hard-coded Modality-Specific FFN architecture, where attention layers are shared across all modalities while FFN layers are fully separated with static routing by modality type, and validate its effectiveness through toy-scale experiments (10-12M parameters). Starting from Vision (Moving MNIST) + Text (TinyStories) bimodal training, we progressively extend to Audio (WavJEPA features) + Text bimodal, and finally Vision + Audio + Text three-way simultaneous training, empirically confirming: (1) Each modality's loss converges independently with no cross-modality interference, even as the number of modalities increases. (2) Separated FFN consistently outperforms both Shared FFN and Soft MoE (Top-1 Gating) across modality types. (3) Catastrophic forgetting manifests as a transient spike followed by full recovery, with multimodal training ultimately improving individual modality performance. (4) Cross-modal attention emerges naturally without explicit supervision. Model and code: https://huggingface.co/2264K/trimodal-hardcoded-moe-toy