Scalable Sparse Transformer Accelerator With In-Memory Butterfly Zero Skipper and Local Attention Reusable Engine for Semi-Structured-Pruned NN

Transformer-based large language models (LLMs) have achieved unprecedented advances across diverse AI tasks. However, their execution remains power-hungry, primarily due to the rapidly growing model size and the quadratic complexity of the self-attention. In-memory computing (IMC) has emerged as a promising solution for accelerating LLMs, delivering high storage density, wide memory bandwidth, and intrinsic compute parallelism. To further improve energy efficiency, this paper addresses LLM inference by leveraging irregular sparse computing in digital SRAM-based IMCs. The proposed butterfly-network-based feed-forward IMC engine enables irregular zero-weight skipping while sustaining high compute utilization. In parallel, a local-attention-reusable IMC engine supports variable sparse attention spans and eliminates redundant self-attention operations. Finally, we implement a four-chip system to scale the parallel performance of the IMC chip. Extensive experiments are deployed on the system, including BERT, GPT to Llama models on 11 AI tasks. The results show that the fabricated accelerator in 28nm CMOS achieves a peak throughput of 186.9 tokens/s and an energy cost of 0.21 J/token, yielding 4.2× speedup and 27.6× energy efficiency over the A100 GPU. Compared to the existing LLM-specific accelerators, the proposed accelerator further achieves 1.2× to 16.9× energy efficiency.