"Just Asking Questions": Doing Our Own Research on Conspiratorial Ideation by Generative AI Chatbots

Interactive chat systems that build on artificial intelligence (AI) frameworks are increasingly ubiquitous and embedded into search engines, Web browsers, and operating systems, or as standalone websites and apps. Researcher efforts have sought to understand the limitations and potential for harm of chatbots powered by generative AI, which we contribute to here. Conducting a systematic review of seven AI-powered chat systems (ChatGPT 3.5 Turbo; ChatGPT 4 Mini; Microsoft Copilot; Google Gemini Flash 1.5; Perplexity; and two versions of Grok), this study examines how these leading products respond to questions related to conspiracy theories. This work is inspired by the “platform policy implementation audit” approach established by Glazunova et al. (2023). We selected five well-known and comprehensively debunked conspiracy theories and four emerging conspiracy theories that relate to breaking news events at the time of data collection. Our findings demonstrate that the promotion of, or pushback against, conspiratorial ideas differ markedly, depending on the chatbot model and conspiracy theory. Our observations indicate that safety guardrails put in place by AI companies are often very selectively designed: appearing to focus especially on ensuring that their products are not seen to be racist; they also appear to pay particular attention to conspiracy theories that address topics of substantial national trauma such as 9/11 or relate to well-established political issues. Future work should include an ongoing effort extended to other chatbots, multiple languages, and a range of conspiracy theories extending well beyond the US.