Does Vision Still Help? Multimodal Translation with CLIP-Based Image Selection

Read full text
Authors: Deepak Kumar, Baban Gain, Kshetrimayum Boynao Singh, Asif Ekbal
Cited by: 0

Abstract

Multimodal Machine Translation aims to enhance conventional text-only translation systems by incorporating visual context, typically in the form of images paired with captions. In this work, we present our submission to the WAT 2025 Multimodal Translation Shared Task, which explores the role of visual information in translating English captions into four Indic languages: Hindi, Bengali, Malayalam, and Odia. Our system builds upon the strong multilingual text translation backbone Indic-Trans, augmented with a CLIP-based selective visual grounding mechanism. Specifically, we compute cosine similarities between text and image embeddings (both full and cropped regions) and automatically select the most semantically aligned image representation to integrate into the translation model. We observe that overall contribution of visual features is questionable. Our findings reaffirm recent evidence that large multilingual translation models can perform competitively without explicit visual grounding.

Fields of Study & Subjects

Multimodal Machine Learning Applications Generative Adversarial Networks and Image Synthesis Natural Language Processing Techniques

Publication details

Publication date
2 January 2026
Document type
Preprint