ConSignformer: Conformer-Based Framework for Continuous Sign Language Recognition

Continuous Sign Language Recognition (CSLR) is a challenging computer vision task that automatically recognizes sequences of sign language gestures from continuous video streams, aiming to bridge communication barriers for the deaf community. Conventional deep learning frameworks for CSLR consist of a single or multimodal feature extractor, a sequence-learning module and a decoder for outputting glosses. The sequence learning module is a crucial part wherein transformers have demonstrated their efficacy in the sequenceto- sequence tasks. However, experimentation in the sequencelearning component for sign language is limited. In this work, we present an innovative CSLR framework, ConSignformer, which combines the strengths of a hybrid network built with S3DCNN and a Conformer-based sequence-learning backbone. To enhance the sequence learning capabilities of the Conformer module, we propose a novel unsupervised pretraining strategy called Regressional Feature Extraction, which uniquely leverages sign pose videos as training data and employs Mean Squared Error loss. We then introduce Cross-Modal Relative Attention (CMRA), which further enhances the ability of the Conformer to learn and utilize complex relationships within the RGB and keypoints data. The experimental results confirm the effectiveness of the adopted pretraining strategy and demonstrate how CMRA contributes to the recognition process. Remarkably, by leveraging a Conformer-based backbone, our CSLR model achieves state-ofthe-art performance on the benchmark datasets: PHOENIX-2014 and PHOENIX-2014T.