Evaluating LLMs’ abilities to create charts, a systematic approach

The use of generative models, especially those based on pretrained transformers, has become a common practice in code development. Tools such as GitHub Copilot, Cursor, and the direct use of conversational chatbots have proven useful to accelerate the development of applications. Unfortunately, generative models are unable to determine what is correct or wrong, and their outputs may contain errors. Their stochastic nature does not guarantee a single solution for the same problem, either. Furthermore, the output depends largely on the prompt issued by the user. To assess the capabilities of LLMs, some benchmarks have been proposed. Unfortunately, they often rely on ground truth data that may not be available. As a result, the extent to which modern LLMs can create charts needs further investigation. This work contributes to the understanding of the generative models’ ability to create charts in three ways: (a) Creating a dataset of prompts, data sources, and chart types to analyze, (b) Designing a set of systematic experiments that cover a wide range of commonly used charts, and variations of the visual variables, and (c) by empirically analyzing the performance of a large set of LLMs of different sizes, including Claude, CodeLlama, Gemini, Gemma, GPT4o, Llama 3.1, and Mixtral. Our results indicate that even the most advanced LLMs have room for improvement.