Multimodal Cosine Similarity Transformer for Gloss-Guided Sign Language Recognition

Continuous sign language recognition (CSLR) requires fine-grained alignment between visual sequences and gloss annotations under weak supervision, which is challenged by modality heterogeneity and ambiguous frame-to-gloss correspondence. We propose a Multimodal Cosine Similarity Transformer (MMCST) to address these issues. MMCST integrates RGB and keypoint heatmap features via gated fusion, and aligns them with gloss embeddings through a Gloss-Conditioned Cosine-Normalized Attention (GCNA) mechanism that stabilizes cross-modal alignment. To further enhance semantic consistency, we introduce Gloss-aware Contrastive Regularization (GLCR). The fused representation is modeled by a cosine-similarity Transformer and decoded with CTC. Experimental results show that MMCST achieves consistent improvements over strong baselines, and ablation studies confirm the effectiveness of gated fusion, GCNA, and GLCR in improving semantic alignment and yielding smoother training dynamics.