Assessing Hallucination Risk in Accessible Image Captions

View at publisher Summarise with VIRAT
Authors: Rostyslav Zatserkovnyi, Zoriana Novosad, R. S. Zatserkovna
Cited by: 0
References: 17

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

The recent advancement of large vision-language models (VLMs) has opened new possibilities for automatic image captioning, offering significant potential for improving web accessibility. However, these models are prone to hallucinations - generating descriptions of objects or details that are not actually present in the source images. Hallucinations represent a critical risk to visually impaired users who rely on these captions as their primary means of understanding visual content, which can lead to confusion and erode trust in assistive technology. In this article, we present a comprehensive evaluation of four vision-language models on the POPE benchmark for hallucination detection, comparing frontier API-based models (Claude Sonnet 4.5 and Claude Haiku 4.5) against open-source models suitable for local deployment (Moondream 2B and Gemma 3 4B). Our evaluation spans 12,000 samples across all four models. Claude Sonnet 4.5 achieves state-of-the-art accuracy of 89.0 % with an F1 score of 0.881, while Moondream 2B offers the lowest risk of hallucination at 1.7 %. We find that local models exhibit a significant precision-recall tradeoff, achieving precision above 96 % but recall of only approximately$\mathbf{5 0 \%}$. Analysis of error patterns reveals that “person” is the most commonly missed object category with 536 instances, and “cat” is the most frequently hallucinated object. We discuss the implications of these findings for accessibility applications.

Fields of Study & Subjects

Multimodal Machine Learning Applications Advanced Neural Network Applications Image Retrieval and Classification Techniques

Publication details

Pages
1–4
Publication date
24 February 2026
Language
EN