SMoLPU: 122.1µJ/Token Sparse MoE-Based Speculative Decoding Language Processing Unit with Adaptive-Offload NPU-CIM Core

SMoLPU is an energy-efficient MoE-based speculative decoding LLM processor with an NPU-CIM core. It has 3 features: 1) Token-adaptive expert refinement removes redundant expert activations and schedules expert load order, achieving$2.3 \times/4.2 \times$energy efficiency improvement in prefill/decode; 2) An adaptive-offload NPU-CIM core and a top scheduling unit maintain HW utilization under dynamic INT–FP ratios, achieving$3.3 \times$lower latency; 3) A reconfigurable DRAM-based LUT-CIM reduces adder tree power/area, while supporting dynamic input size. The LLM processor achieves 43.5% lower energy per parameter than prior SOTA.