Patch-Range Transformer (PRT): Efficient Local Attention for Vision Transformers

Read full text Summarise with VIRAT
Authors: Bala Murugan
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Patch-Range Transformer (PRT): Efficient Local Attention for Vision Transformers Vision Transformers (ViT) have emerged as a powerful architecture for image classification by leveraging global self-attention mechanisms. However, their reliance on full pairwise attention results in quadratic computational complexity with respect to the number of image patches and limits their ability to effectively capture local spatial structures. This leads to increased computational cost and reduced data efficiency, particularly when training on small to medium-scale datasets. In this work, we propose the Patch-Range Transformer (PRT), a simple yet effective modification to the standard Vision Transformer that introduces spatially constrained local attention without increasing model complexity. Specifically, each patch token attends only to its neighboring patches within a fixed Chebyshev distance on a two-dimensional patch grid. Meanwhile, a global classification (CLS) token maintains full attention across all patches, enabling efficient global information aggregation and preserving long-range dependencies. Unlike prior approaches such as Swin Transformer, which rely on hierarchical architectures, shifted window mechanisms, and additional parameters, PRT achieves efficient local attention through a lightweight masking strategy. This design introduces no additional parameters, requires minimal architectural changes, and can be seamlessly integrated into existing ViT frameworks. The key contributions of this work are as follows. First, we introduce a Patch-Range attention mechanism based on Chebyshev distance for efficient spatially local interactions. Second, we propose a CLS-based global communication strategy that enables cross-region information flow without complex architectural modifications. Third, we demonstrate a parameter-efficient design that maintains identical model size while reducing attention FLOPs by up to 61%. Finally, we provide a comprehensive ablation study showing that a moderate attention range yields the optimal balance between accuracy and computational efficiency. Extensive experiments on benchmark datasets, including CIFAR-10, CIFAR-100, and Caltech-101, are conducted under identical training conditions without pretrained weights. The results show that PRT consistently outperforms the standard ViT while maintaining the same parameter count. On CIFAR-10, PRT achieves 82.11% accuracy, improving over ViT by 1.00% while reducing FLOPs by 61%. On CIFAR-100, it surpasses both ViT and Swin Transformer while using approximately ten times fewer parameters than Swin. On Caltech-101, PRT achieves significant accuracy gains over both baselines. An ablation study on the attention range parameter further reveals that R = 2 achieves the best trade-off between computational efficiency and classification performance, attending to only a subset of patches while maintaining optimal accuracy. These findings highlight the effectiveness of structured local attention in improving efficiency without sacrificing model performance. Overall, the Patch-Range Transformer provides a scalable, parameter-efficient, and computationally optimized alternative to global attention mechanisms in Vision Transformers. Its simplicity, effectiveness, and strong empirical performance make it well-suited for real-world applications and resource-constrained environments. Code and resources:GitHub: https://github.com/Balamurugan245/Patch-Range-TransformerKaggle: https://www.kaggle.com/code/balamuruganaiml/brain-tumor-mri-prt

Fields of Study & Subjects

Advanced Neural Network Applications Visual Attention and Saliency Detection Domain Adaptation and Few-Shot Learning

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
28 March 2026
Language
EN