Adaptive Gated Fusion of Visual and Skeletal Dynamics: A Mamba-Based Architecture for Sign Language Recognition

Sign Language Recognition (SLR) necessitates capturing complex spatio-temporal dependencies; however, Transformer-based models often suffer from quadratic complexity and treat video frames as discrete token sequences, potentially limiting their ability to capture the fluid dynamics of sign language. To address this, we introduce a novel multi-modal hybrid architecture leveraging the MambaVision backbone, which utilizes a selective scan mechanism to efficiently model visual content. This architecture enables content-aware feature extraction, dynamically prioritizing discriminative motion cues over static background information without the computational overhead of global self-attention. This work marks one of the first applications of State Space Models (SSMs) to word-level SLR. We propose the Adaptive Gated Feature-wise Concatenation Layer (AG-FiCL), a non-linear fusion mechanism designed to intelligently modulate visual features using skeletal context. By leveraging a gated scaling mechanism and concatenative shifting, AG-FiCL effectively overcomes the limitations of standard linear conditioning methods, such as FiLM, in capturing intricate pose-appearance relationships. Validated on three diverse benchmarks, our model demonstrates superior representation learning capabilities, particularly in challenging signer-independent scenarios. We achieve highly competitive accuracy of 99.53% on LSA64, 97.05% on BosphorusSign22k, and 98.77% on the INCLUDE dataset. These results establish that Mamba-based architectures, enhanced with context-aware fusion, offer a robust and scalable paradigm for modeling the intricate dynamics of sign language compared to conventional CNN and Transformer baselines.