TFLOP: Towards Energy-Efficient LLM Inference An FPGA-Affinity Accelerator with Unified LUT-based OPtimization

Large Language Models (LLMs) suffer from significant performance and energy efficiency bottlenecks during the memory-bound decoding stage, where GPUs are often underutilized. We propose TFLOP, a novel CPU-FPGA heterogeneous prototype system that addresses this challenge by employing a 4-bit product quantization scheme on model weights and the KV cache. This approach decomposes GEMV operations in decoding stage into two hardware-friendly steps: centroid reconstruction and table lookup, which are efficiently mapped onto an FPGA’s heterogeneous resources. Our key innovation is a unified FPGA architecture that can handle both row- and column-wise quantization, simplifying hardware design and improving efficiency. Evaluations show that TFLOP achieves superior performance, delivering a 2.76$\times$ speedup over the NVIDIA A100 GPU on the LLaMA-2-7B model, while maintaining high accuracy and exceptional energy efficiency.