AHQ: Adaptive HyperNetwork Quantization for Edge Deployment of Large Language Models

Scaling up large language models (LLMs) to run on resource-limited devices, including mobile phones and embedded systems, is greatly restricted by their high size and computational requirements, which prevent them from being practically deployed in edge settings. To solve this, we present Adaptive HyperNet-Quant (AHQ), a new quantization method that employs adaptive, hypernetwork-controlled bit-width assignment. AHQ utilizes tiny, input-dependent hypernetworks embedded within every layer of the LLM, combined with quantization-aware training (QAT), to dynamically compute optimal quantization parameters, such as bit-widths and scaling factors, in real-time. Our experimental results show considerable model size reduction, large inference speedup, and preserved or even enhanced accuracy over static quantization techniques. These highlight AHQ's potential to enable the efficient deployment of high-performance LLMs on edge devices, unlocking a wide range of applications in offline and resource-Limited environments.