Multimodal Image Captioning with Transformer-based Dual Attention

Image captioning is the process of automatically generating natural language descriptions to understand the image content by integrating visual understanding and language generation. Recent developments in transformer based architectures have improved caption clarity and contextual understanding, still most existing methods depend only on visual features and ignore textual features present in the real world images. Scene text includes signs and labels within images that provide important semantic information that can help image understanding during learning. This paper presents a multi-modal image captioning system that combines visual features extracted using a Vision Transformer(ViT), scene text features extracted through Optical Character Recognition(OCR) and encoded using Bidirectional Encoder Representations from Transformers (BERT) for contextual textual representation. The visual features and textual representations are fused using an adaptive multi-modal feature fusion method to handle noisy or missing text information and captions are generated using a transformer based decoder with attention mechanisms. Experiments are conducted on Flickr30k dataset and quantitative evaluation is done using standard metrics such as BLEU, METEOR and CIDEr. The proposed approach aims to improve caption accuracy and semantic depth by effectively integrating information from multiple modalities, making it useful for real world image understanding applications where both visual content and scene text play a critical role.