Edge-Optimized Vision Transformers: A Co-Designed Hardware-Software Framework for Efficient Attention Mechanism

Vision Transformers (ViTs) effectively capture global context in videos/images using attention mechanisms, outperforming the traditional Convolutional Neural Networks (CNNs). However, the attention mechanism's dependence on multiple matrix multiplications results in high computational complexity, posing a significant challenge for edge computing devices. This paper proposes a hardware-software co-design approach to mitigate this complexity. The software part utilizes a 50% sparsity pattern, which doubles throughput, and replaces the softmax function with linear Taylor attention for enhanced hardware efficiency. The hardware design introduces Recursive Array Matrix Multiplier (RAMM), a new architecture tailored for ViTs, which outperforms the conventional Systolic Array Matrix Multiplier (SAMM) used in similar works. RAMM is applicable to any deep neural network (DNN) architecture, but since ViTs are recent models, to demonstrate the versatility of RAMM, we integrate it into two ViT variants: the original ViT and MaxViT, a hybrid model that incorporates convolution. In addition, a Sparse Recursive Array Matrix Multiplier (SpaRMM), is also designed, which leverages 2:4 structured sparsity to achieve additional reductions in processing units and power consumption. The proposed architectures are synthesized using the AMS 90 nm library at a clock frequency of 500 MHz and also implemented on FPGA. The SpaRMM architecture achieves superior latency and throughput compared to RAMM, with increasing efficiency for higher-order matrices. Furthermore, RAMM demonstrates significant improvements over matrix multipliers in state-of-the-art co-design frameworks, such as SpAtten and$A^{3}$.