From Generation to Detection: Multimodal Generative AI and the Threat of Automated Misinformation

Generative AI is advancing rapidly, raising growing concerns about its role in spreading misinformation. In this study, conducted in collaboration with a media house, we investigate how multimodal AI systems (text+image) could be misused to create and promote misleading news content. Our aim is not to endorse such misuse, but to identify risks so that stronger safeguards can be developed. We simulate realistic scenarios where malicious actors generate deceptive news images using five variants of Stable Diffusion, including advanced models such as Stable Diffusion 3 and Stable Diffusion with Direct Preference Optimization (DPO), across three well-known misinformation datasets: Snopes, Pheme, and Politifact. We evaluate how convincing these AI-generated images are by measuring their alignment with news text, and then assess whether GPT-4 can reliably classify and rank them compared to human judgment. Beyond testing risks, we also analyze whether generative AI behaves differently with legitimate (non-rumour) content. Our findings reveal both the power and the limitations of current AI in misinformation pipelines. Importantly, they highlight the need for stronger safeguards, transparent monitoring, and human oversight to prevent AI from amplifying false narratives. This work provides actionable insights for policymakers, media watchdogs, and developers working toward responsible AI deployment and the protection of information integrity in society.