Gated fusion and multi-task learning for multimodal emotion recognition in meme-rich contexts

Multimodal emotion recognition plays a vital role in social media analysis. However, traditional systems face a fundamental challenge: cross-modal semantic conflicts triggered by internet memes lead to a severe negative transfer effectsignificantly degrading the model's ability to recognise emotions even in regular non-meme content. This study systematically reveals and quantifies this phenomenon, providing critical insights for overcoming the bottleneck of multimodal recognition in meme-rich contexts. To investigate this issue, we constructed a balanced mixed dataset and first empirically confirmed that conventional fusion methods suffer a substantial performance drop under this setting. In response, we propose the Meme-Adaptive Gated Fusion Network (MAG-Net), which is built upon shared BERT-ResNet encoders, incorporates a gated fusion module to dynamically adjust the fusion weights of image and text features, and employs a multi-task learning framework to jointly optimise emotion classification and meme detection. Through feature sharing and the adaptive modulation of fusion ratios via the gating network, our method effectively mitigates cross-modal conflicts. Our experiments demonstrate that on a test set containing 50% memes, the method achieves 56.93% emotion recognition accuracy, outperforming the baseline by 14.65%, while reaching 94.02% meme detection accuracy. Importantly, our framework not only improves overall performance but also restores the model's ability to correctly process regular content, thereby recovering the advantages of multimodal fusion in challenging meme-rich scenarios.