HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models (Student Abstract)
Abstract
Vision-Language Models (VLMs) encode images into lengthy sequences of visual tokens, leading to excessive computational overhead and limited inference efficiency. In this paper, we study the hierarchical attention pattern in vision encoders and propose HiPrune, a training-free and model-agnostic token Pruning framework for VLMs. We identify that middle layers in the vision encoder attend to object-centric regions, while deep layers capture global contextual features. Based on this observation, HiPrune selects tokens based on the attention score from the middle and deep layers. Our method requires no retraining and integrates seamlessly with any ViT-based VLM. Experiments demonstrate that HiPrune achieves outstanding pruning performance, maintaining a balance between efficiency and efficacy.
Fields of Study & Subjects
Multimodal Machine Learning Applications
Generative Adversarial Networks and Image Synthesis
Domain Adaptation and Few-Shot Learning
Publication details
- Journal / source
- Proceedings of the AAAI Conference on Artificial Intelligence
- Publisher
- Association for the Advancement of Artificial Intelligence (AAAI)
- Volume
- 40
- Issue
- 48
- Pages
- 41275–41277
- Publication date
- 14 March 2026
- Language
- EN