Demystifying the Cost Versus Benefits of Sparse Large Language Model Acceleration

The rapid growth in LLM model sizes has intensified the need for sparsification techniques to reduce computational and memory overheads. This presents a unique opportunity to design accelerator chips that can exploit sparsity to improve runtime efficiency. Supporting sparse execution in AI accelerators requires coordinated enhancements across three key subsystems: compute, memory, and interconnect. However, there is a lack of systematic methodologies that provide architectural insights for building efficient sparse accelerators tailored to diverse sparsity patterns and model architectures. In this work, we systematically analyze trade-offs between area, energy, and performance under quality constraints across a broad range of AI models. We quantify the architectural impact of enabling sparsity on key hardware subsystems, and characterize how design choices in compute, memory, and interconnect affect overall efficiency. Finally, we explore the design space of sparse accelerators and identify optimal configurations for accelerating modern LLM workloads under practical constraints.