A Confidence-Aware Human–Machine Collaboration Framework for Harmful Meme Detection

With the rapid advancement of technology, social media has become a dominant platform for expressing human emotions such as joy, anger, sorrow, and happiness. Among these expressions, memes serve as a powerful yet double-edged medium of communication—capable of spreading humor and creativity but also amplifying harmful or misleading messages. Although many studies have explored harmful meme detection, few have incorporated deferral mechanisms into this task. While machine learning models effectively handle large-scale multimodal data, they often struggle to interpret context-dependent or ambiguous content where human judgment is crucial. To address this limitation, this article proposes a Confidence-Aware Human–Machine Collaboration Framework that dynamically allocates classification tasks based on model confidence, introducing a deferral strategy that delegates low-confidence samples to human experts while leveraging proxy labeling and a fusion model to refine decision quality. Experiments using three pretrained models (ExplainHM, IntMeme, and LLaVA) across five benchmark meme datasets demonstrate that the proposed framework substantially improves classification accuracy, particularly in high-uncertainty cases, highlighting the effectiveness of human–machine collaboration in harmful meme detection. By enabling reliable moderation of harmful visual content, the proposed framework contributes to building safer and more trustworthy online ecosystems. It supports social platforms in mitigating misinformation, hate speech, and manipulation, thereby enhancing the integrity of digital communication.