A Structured-Sparsity-Based Design Approach for Energy-Efficient Spiking Transformer Processing

View at publisher Summarise with VIRAT
Authors: Hyunseok Jung, Kyungchul Lee, S. Park
Cited by: 0
References: 18

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Spiking transformers (STs) have emerged as promising architectures that achieve competitive accuracy with artificial neural network (ANN) on large-scale datasets. Despite this progress, the efforts to reduce the computational complexity of spiking self-attention (SSA) have seldom been explored. In this work, we present sparsity-based design approaches to reduce the computational complexity of SSA execution by systematically exploiting the structured sparsity in SSA. The proposed approaches are based on the observation that SSA naturally exhibits structured sparsity, which can be exploited to identify and skip redundant computations in SSA blocks. First, by employing a novel fully spiking SSA operator (FSSA) incorporating additional leaky integrate-and-fire (LIF) neurons, the average structured sparsity of SSA has been increased by$1.8\times $with less than 1% loss in accuracy. In addition, by adopting a filtering strategy that ignores neurons with low spike rate when detecting structured sparsity, additional$1.5\times $structured sparsity has been achieved compared to FSSA with negligible accuracy loss. Then, a sparsity-aware dataflow and hardware design convert the proposed structured sparsity patterns into runtime skipping of computations and weight transfers in SSA block execution. In postsynthesis simulations across the evaluated models, the proposed SSA operator with filtering techniques improve SSA block effective throughput and energy efficiency to 234.44 GOP/s and 319.12 GOP/J while keeping model accuracy loss below 1%, corresponding to 24.3% and 28.2% gains over the baseline spiking neural network (SNN) accelerator.

Fields of Study & Subjects

Advanced Memory and Neural Computing Advanced Neural Network Applications Low-power high-performance VLSI design

Publication details

Journal / source
IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Volume
34
Issue
7
Pages
2064–2073
Publication date
25 March 2026
Language
EN