BLAST: Blockwise Activation Sparsity for Transformers

Transformer-based vision models achieve state-of-the-art accuracy but incur high inference costs due to dense matrix operations. Prior work has explored activation sparsity to reduce computation, but largely produces unstructured patterns, which are difficult to exploit on modern hardware such as recent NVIDIA GPUs, which provide native acceleration for semi-structured formats like 2:4 sparsity. We presentBLAST, a simple and effective approach for inducing 2:4 activation sparsity in transformers via a novel blockwise extension of the squared Hoyer norm. Applied during fine-tuning,BLASTincreases MLP activation sparsity from ∼50% to over 91%, while maintaining accuracy across standard image benchmarks. Crucially, the resulting structured sparsity conforms to the 2:4 format supported by NVIDIA GPUs , yielding real-world inference speedups of up to 1.34× in the MLP layers and up to 1.15× overall. By aligning activation sparsity with hardware constraints,BLASTmakes sparsity-based acceleration feasible in real-world deployments.