Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning

Read full text
Authors: Association for Artificial Intelligence 2026, Qiang Huang, Lingyu Li, Xiaopeng Liu, Xiaoxing You, Jun Yu, Min Zhang, Zhang, Chi
Cited by: 0

Abstract

News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce \textbf{MERGE}, the first \textbf{M}ultimodal \textbf{E}ntity-aware \textbf{R}etrieval-augmented \textbf{GE}neration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.

Fields of Study & Subjects

Multimodal Machine Learning Applications Image Retrieval and Classification Techniques Topic Modeling

Publication details

Journal / source
Underline Science Inc.
Publication date
7 January 2026
Language
EN