Uncertain Aspect-Aware Triplet Disentangling-Transferring for Domain-Specific Multimodal Neural Machine Translation

Domain-specific multimodal neural machine translation (DMNMT) aims to incorporate text-related visual details into text to enhance machine translation performance in specific domains. Existing DMNMT approaches often borrow cross-modal fusion strategies from multi-modal neural machine translation (MNMT). However, these DMNMT methods rely heavily on their corresponding visual scene information, resulting in a lack of robustness in translating domain-specific expressions. This issue, known as the “domain semantic shift problem,” is particularly evident in noisy scene scenarios and even text-only scenarios. To this end, this paper adopts an Uncertain Aspect-aware Triplet Contrastive Transferring strategy (UAs-TriCT) through a cross-modal regularized variational disentangling-distillation mechanism for DMNMT. Our proposed approach enhances the model's ability to generate semantically aligned target sentences while preserving sensitivity to domain-specific aspects. Specifically, we propose a regularized variational disentangling-distillation strategy to capture modality- and linguistic-shared Uncertain Aspect-aware (UAs) representations. This mechanism can generate modality-shared virtual visual representations from the text through two types of regularized distillation, even in noisy or text-only scenarios. Then, we employ an UAs-TriCT strategy to ensure cross-modal and cross-linguistic semantic alignment while maintaining UAs-center awareness during target sentence generation. We conduct experiments on four benchmark MNMT datasets, covering domain-specific, multi-domain, and general-domain scenarios. The experimental results demonstrate that our proposed approach achieves state-of-the-art performance across all test sets for both MNMT and DMNMT datasets. The in-depth analysis further validates the effectiveness of the disentangling-distillation and UAs-TriCT strategies across aligned-image, noisy-image, or even text-only scenarios.