Do Multimodal LLMs Pay Attention to What They Actually See?

Reasoning abilities of LLMs are a prominent issue nowadays, and much work has been done to compare and analyze differences between human and model logical step sequences. Symbolic benchmarks have generally assessed this question in the context of solving mathematical and programming problems, but there remains a gap in evaluating LLM performance on visually driven tasks. Based on the symbolic Re-Imagine solution, a Re-Imagine Vision benchmark is proposed to examine model performance on image classification with logical manipulations. Primary outcomes on open-source variants show that MLLMs perform well at the perception stage and easily ignore visual noise but struggle to disregard irrelevant information in tasks that require deeper reasoning levels and the construction of associative chains. This highlights a weakness in the attention mechanisms used in MLLMs architecture.