On VLMs for Diverse Tasks in Multimodal Meme Classification

The use of multimodal memes to spread hatred, propaganda, and violence across social and digital media necessitates effective content moderation, which can be addressed through AI-based meme analysis. In this paper, we present a comprehensive and systematic analysis of Vision-Language Models (VLMs) for disparate meme classification tasks, and introduce a novel approach: Combining VLM Explanation to Fine-tune LLMs (CoVExFiL). In the proposed CoVExFiL, we generated a VLM-based understanding of the meme images and used this information to fine-tune Large Language Models (LLMs) based on the embedded meme text. Our contributions are threefold: (1) Benchmarking VLMs using diverse prompting strategies for these sub-tasks; (2) Evaluating LoRA fine-tuning across all VLM components to assess performance gains; and (3) Proposing the novel CoVExFiL approach, where detailed meme interpretations generated by VLMs are utilized to train smaller language models (LLMs), thereby significantly improving classification. Following extensive experimentation, we observed that CoVExFiL improved the baseline performance by 8.34%, 3.52%, and 26.24% for sarcasm, offensive content, and sentiment classification, respectively. These findings shed light on the capabilities and shortcomings of VLMs, while also establishing CoVExFiL as a promising strategy for advancing meme understanding. The code is available at https://github.com/gavit21/Memes-Understanding-with-VLMs https://github.com/gavit21/Memes-Understanding-with-VLMs. CAUTION: This paper may contain harmful content.