DHE: A Semantic-Preserving Framework for Robust Post-Training Quantization of Vision Transformers

Although Vision Transformers (ViTs) have demonstrated impressive achievement in computer vision, they suffer from considerable flaws in deployment due to high computational and memory costs. Post-training quantization (PTQ) is an effective compression method but can cause severe accuracy failure of ViTs, which is mainly caused by the disturbance of attention mechanisms. Based on the scheme of fixed threshold suggested by Zhenhua Liu et al., this paper addresses this drawback by introducing a Dynamic Hybrid Enhancement (DHE) scheme, which changes the quantization paradigm of numerical reconstruction to that of semantic preservation. The main innovations are: the dynamic mechanism of adjusting the loss of rankings by dynamically moving the attention through distribution; the sensitivity of the weight matrix to differences, which prioritizes the semantically important attention connections; and the multi-head normalization strategy, which optimizes attention heads. Numerous experiments on CIFAR-10 and CIFAR-100 show that DHE has accuracy rates of 67.22% and 37.62% compared to the baseline PTQ model (i.e., the fixed-threshold method by Zhenhua Liu et al.), which is 1.82 and 2.23. The role of every single component is confirmed with the help of ablation studies, and the performance of semantic preservation through attention visualization and quantitative measures (e.g., Attention Ranking Preservation Rate, ARPR = 94.7% on CIFAR-10, ARPR = 90.1% on CIFAR-100) proves the superiority of the suggested pattern to the traditional ones.