Area- and Utilization-Efficient LLM Accelerator With Fused Speculative Decoding for Edge-Side Inference

The inference of large language models (LLMs) on edge devices has always been a challenge, the autoregressive inference and enormous number of parameters result in long inference latency. Although speculative decoding is proposed for inference acceleration, it poses challenges for hardware accelerator design, as the differing computational characteristics of the drafting and verification stages make it difficult to optimize both chip area and hardware utilization. This brief introduces fused speculative decoding (FSD) to optimize LLM inference on edge devices by unifying all the operations to general matrix multiplications during inference. The proposed FSD-Infer algorithm fuses the drafting and verification phases of conventional speculative decoding and enables weight sharing, reducing off-chip memory access, and inference latency without requiring retraining or fine-tuning. For hardware, we introduce FSD-Acc, an area- and utilization-efficient hardware accelerator that efficiently executes the fused operations enabled by FSD-Infer. Experimental results show that compared with autoregressive inference, FSD-Infer reduces EMA by up to 26.11%, enhances arithmetic intensity by$10.26\times $, and speeds up GPU inference by up to$1.38\times $. When deployed on Xilinx ZCU102 FPGA board at 200 MHz working frequency, FSD-Acc achieves the best area efficiency and energy efficiency, outperforming the state-of-the-art LLM accelerator for edge inference by$2.53\times $(TPS/kDSP),$5.90\times $(TPS/kLUT) and$2.02\times $, respectively.