Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

Read full text Summarise with VIRAT
Authors: Mincheol Kwon, Minseung Lee, Sol-ah Choi, Miso Choi, Kyeong-Jin Oh, Hyunyoung Lee, CheonYoung Park, Yongho Song, Seunghyun Park, Jinkyu Kim
Cited by: 0
References: 34

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.

Fields of Study & Subjects

Multimodal Machine Learning Applications Handwritten Text Recognition Techniques Advanced Neural Network Applications

Publication details

Journal / source
arXiv (Cornell University)
Publication date
24 March 2026
Language
EN
Document type
Preprint