Unified Multimodal Architecture for Accessible Video Understanding: Integrating Vision, Captions, Sign Language, and Audio Descriptions with Robustness to Modality Degradation

Video accessibility remains a critical challenge for people with visual and hearing impairments, despite the exponential growth of video content. Current accessibility solutions operate in isolation—audio descriptions for the blind and low vision (BLV) community, captions for the deaf and hard of hearing (DHH) community, and sign language interpretation as a separate overlay—missing opportunities for synergistic multimodal understanding. This paper introduces UMA (Unified Multimodal Architecture), an end-to-end framework that integrates vision, captions, sign language, and audio descriptions into a cohesive system designed for accessible video comprehension. The core innovation lies in parameter-efficient robustness to modality degradation: leveraging intermediate feature modulation with fewer than 0.7% additional parameters, UMA maintains comprehension fidelity even when individual modalities are corrupted, delayed, or missing—a realistic constraint in real-world video streaming and broadcasting environments. We validate UMA across 40,000 videos using rigorous evaluation with 347 sighted participants, 40 BLV users, and 7 professional audio describers, demonstrating that unified multimodal descriptions significantly outperform isolated accessibility services (p