Protecting multimodal large language models against misleading visualizations

Visualizations play a pivotal role in daily communication in an increasingly data-driven world.Research on multimodal large language models (MLLMs) for automated chart understanding has accelerated massively, with steady improvements on standard benchmarks.However, for MLLMs to be reliable, they must be robust to misleading visualizations, i.e., charts that distort the underlying data, leading readers to draw inaccurate conclusions.Here, we uncover an important vulnerability: MLLM question-answering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline.To address this, we provide the first comparison of six inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones.We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points.We make our code and data available.1 What is the proportion of death of coronavirus as a proportion of the number of cases?Around 66%Around 16% Around 6%Around 36%What was the general trend in gun deaths in Florida from 2003 to 2007?then Cannot be inferred then Were there more abortions than cancer screenings in 2011?Yes Cannot be inferred No MisrepresentationThe numerical values are not proportional to the height of the bars. Dual axisCancer screenings and abortions are shown on two different axes.