An embedded vision transformer with quantization-friendly simple attention, optimized buffers, and SIMD acceleration

Vision Transformer (ViT) is the key to state-of-the-art DNN models in computer vision. Due to the high memory requirement and computational burden of ViTs, a lightweight design and integer quantization are essential approaches . However, these lightweight implementations still target desktop-class hardware and high-end edge devices. The deployment on resource-restricted embedded devices remains a significant challenge. In this context, floating-point arithmetic avoidance, buffer management, and CPU’s SIMD leverage cannot be ignored to meet embedded system requirements, such as memory footprint and real-time execution. Therefore, we implement an ultra-lightweight ViT, i.e., a fully integer CNN-Transformer hybrid model with Simple attention by Post-Training Quantization (PTQ), targeting embedded hardware. Specifically, we introduce Normalization Equivalent Quantization (NEQ) to reduce l-1 normalization computation, and buffer optimization, e.g., Attention row buffer to mitigate $$O(n^2)$$ buffer size increase. As a result, the parameter size can be kept under 512 KiB and can be deployed in on-chip memory on a small-scale RISC-V core, CV32E40p. Additionally, with SIMD extension and its dedicated kernels, our implementation achieves 10.4 $$\times $$ acceleration for the logit of Simple attention, resulting in 1.4 $$\times $$ latency improvement compared with the combination of typical quantization and the cutting-edge Integer ViT module.