Efficient Edge Deployment of Mixture-of-Experts Models With Hybrid Expert Quantization

Deploying large language models (LLMs) directly at the edge offers significant benefits in reducing inference latency and enhancing privacy protection. However, this approach presents critical technical challenges due to the limited hardware resources available at the edge. In this paper, we explore the deployment of the Mixture-of-Experts (MoE) model, a popular LLM architecture, using weight quantization in a typical edge environment with multiple edge servers. Existing methods that uniformly quantize MoE models do not account for the varying importance of different experts or the dynamic nature of edge requests. To address these issues, we propose a hybrid expert quantization strategy, where each edge server quantizes experts to different bit-widths based on the preferences of received requests. Furthermore, we implement a comprehensive system optimization that jointly considers expert quantization and request assignment, aiming to achieve high-precision and low-latency inference services. To manage the complex, coupled optimization challenges, we design a reinforcement learning (RL)-based algorithm. To overcome the slow convergence typical of traditional RL, we pre-train the policy network using a system simulator that generates heuristic solutions, thus accelerating the learning process. Extensive experimental results on real datasets demonstrate that our approach significantly outperforms existing methods.