Do Large Language Models Reason About Uncertainty Like Humans? A Benchmark on Hurricane Forecast Visualization Comprehension

Uncertainty visualizations, such as hurricane cones and ensemble tracks, are vital for risk communication but are frequently misinterpreted, leading to dangerous outcomes. No current visualization reliably conveys forecast uncertainty without cognitive bias. As AI assistants like large language models (LLMs) increasingly support public understanding and decision-making, they offer a promising pathway to enhance the interpretation of such complex visualizations. We present HurricaneTrust, the first benchmark to systematically compare how humans and LLMs reason about hurricane forecast uncertainty visualizations. HurricaneTrust spans two escalating phases and includes seven representative visualization formats, six real hurricane scenarios, and three agent types (humans, LLMs with context, and LLMs without context), resulting in 880 visualizations and over 117,600 structured question–answer pairs under matched evaluation conditions. Phase 1 contrasts LLM and human performance across implicit and explicit uncertainty encodings; Phase 2 examines reasoning under single- versus multi-dimensional uncertainty representations. Our evaluation includes a thorough analysis of damage estimation, reasoning strategies, and comprehension patterns. Results reveal that LLMs have a stronger semantic and conceptual understanding of uncertainty, and are less misled by visual variability, but still replicate key human biases during decision-making. Our findings offer insights into aligning LLM behavior with human cognition in uncertainty-rich visual reasoning tasks.