GDE: Grid-based Diversity Enhancement for Efficient Vision Token Selection in VLMs

Recent vision-language models (VLMs) have achieved impressive performance across various tasks but suffer from high computational cost due to the large number of visual tokens. While diversity-based token selection methods effectively reduce redundancy, they often lose spatial coverage under tight token budgets. To address this issue, we propose grid-based diversity enhancement (GDE), a lightweight module that enforces balanced spatial coverage by ensuring that each grid region contributes at least one representative token. $GDE$ can be seamlessly integrated into existing diversity-based frameworks with minimal overhead. Experiments on image understanding benchmarks using LLaVA-1.5-7B demonstrate that $GDE$ consistently improves performance, achieving $97.4 \%$ of the vanilla model’s accuracy with only 128 tokens. These results show that $GDE$ effectively preserves global visual context and achieves a better trade-off between efficiency and accuracy under extreme compression.