Improving Sign Language-Gloss Translations With Pretrained Models

Sign language gloss-to-text translation is a crucial task for enhancing communication and accessibility for the deaf community. However, the scarcity of annotated sign language data poses significant challenges for developing effective translation models. In this paper, we propose leveraging pretrainedxian language models to improve gloss-to-text translation performance. While intuitively one might assume bilingual translation objectives (e.g., En-De) suit this task, our extensive empirical analysis yields a counter-intuitive but critical key finding: the gloss-to-text task functions fundamentally as a monolingual constraint generation task rather than a cross-lingual translation task. Consequently, models pretrained with denoising auto-encoding objectives (e.g., mBART) significantly outperform those with translation objectives. Based on this insight, we propose to utilize large-scale multilingual pretrained models in our main experiments on three widely-used benchmarks (PHOENIX-2014T, ASLG-PC12, and CSL-Daily). Experimental results demonstrate the superiority of our approach, which consistently benchmarks against and outperforms existing state-of-the-art models. Specifically, our method achieves absolute improvements of +4.37 BLEU on Phoenix2014T and +15.18 BLEU on CSL-Daily compared to strong baselines. We also discuss limitations, such as copying behavior and model robustness, which warrant further investigation. Our findings suggest that denoising auto-encoding pretrained models hold promise for advancing gloss-to-text translation and have the potential to greatly benefit the deaf community.