MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production

Proceedings of the 21st European Conference on Computer Systems 2026 Open access

Read full text
Authors: Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Hua Xie, Qi Huang, Wen Heng, Yiyuan Ma
+8 more Wenlei Bao, Size Zheng, Xuegui Zheng, Yanghua Peng, Haibin Lin, Xin Jin, Xin Jin, Xin Liu
DOI: 10.1145/3767295.3769325
Cited by: 1
References: 56

Abstract

We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware.

Fields of Study & Subjects

Generative Adversarial Networks and Image Synthesis Advanced Neural Network Applications Machine Learning and Algorithms

Publication details

Journal / source
Proceedings of the 21st European Conference on Computer Systems
Publisher
ACM
Pages
366–382
Publication date
24 April 2026
Language
EN