Variational Nearest Neighbor Sign Language Translation

Sign Language Translation (SLT) aims to convert continuous sign language videos into textual sentences. However, sign videos contain intricate semantic information, rendering SLT a challenging task that requires abundant parallel data for effective model training. Therefore, previous studies have typically fine-tuned SLT models using pre-trained general-domain models due to the scarcity of SLT data, aiming to transfer general knowledge to the specific sign language domain. However, the exploitation of SLT data is insufficient, as the training data is eliminated from inference, with only the stored model weights being utilized. Moreover, the notorious long-tail distribution issue is further exacerbated by data scarcity. In this study, we propose a Variational k Nearest Neighbor framework for Sign Language Translation (VK-SLT), which introduces kNN retrieval from a datastore constructed from the training data to enhance the next token prediction during inference. To mitigate the long-tail distribution issue, a pre-trained Variational Auto-Encoder (VAE) is employed as the generative model to build the datastore, resulting in a re-distributed datastore with resampling. Further, the datastore is optimized and compressed through contrastive learning for effective and efficient kNN retrieval. The proposed VK-SLT surpasses the previous state-of-the-art results on two widely used benchmarks, demonstrating the effectiveness of our framework.