KAP: Knowledge-Augmented Prefix Tuning for Image Captioning

Existing lightweight image captioning methods based on pre-trained models often suffer from insufficient cross-modal fusion and the semantic redundancy introduced by external memory. To mitigate these issues, we propose a novel Knowledge-Augmented Prefix tuning model for image captioning (KAP). Specifically, our model first introduces a Hierarchical Feature-Fusion based Keyword Retrieval module to dynamically retrieve relevant keywords from an external memory, effectively avoiding the noise and redundancy typically associated with retrieving full sentences. Subsequently, the proposed Triple-Interactive Prefix Generator leverages a multi-head cross-attention mechanism to seamlessly integrate the retrieved keywords, image features, and learnable prefix parameters, thereby generating a highly condensed knowledge-augmented prefix to steer the frozen language model. Extensive experiments on the MS-COCO, Flickr30k, and NoCaps datasets demonstrate that KAP, with only 85MB of trainable parameters, not only achieves competitive performance compared to fully fine-tuned large models but also significantly outperforms existing lightweight methods, confirming its capability to generate accurate and descriptive captions.