Mixture-of-Experts Multi-View Large Vision-Language Models for Multimodal Fake News Detection

Existing multimodal fake news detection methods based on traditional small models are prone to learn superficial features while struggling to perform deep reasoning and truly perceive fine-grained image-text consistency. Recently, fueled by Large Language Models and large-scale multimodal pretraining data, Large Vision-Language Models (LVLMs) have seen significant progress in these aspects. In our pilot study, domain-specific fine-tuned LVLM (FT-LVLM) Qwen2-VL-2B has achieved superior performance for multimodal fake news detection. However, we still find two weaknesses within it: (1) The model tends to rely only on textual modality while learning little from visual modality. (2) Directly outputting predictions hinders the model from exploiting the powerful reasoning capability of the LVLM. To address the above issues, we further propose a novel framework ofMixture-of-ExpertsMulti-View LargeVisual-LanguageModels (MMVM) to explicitly disentangle multi-view features within the LVLM and adaptively fuse them for fake news detection. Specifically, we first extract representations of three distinct views, i.e.,main branch,visual branchandreasoning branch, from the LVLM respectively. Then, considering that a given piece of fake news may not contain clues of equal importance in all the views, View-Aware Mixture-of-Experts networks (VAMoE) are proposed to dynamically reweight single-view representations and route the most informative features for the final prediction. Extensive experiments on four public datasets demonstrate the significant effectiveness and generalization ability of our proposed framework.