Parallelism Strategies and Concurrency Effects for Mixture-of-Experts Inference on GPU Systems

Mixture-of-Experts (MoE) architectures reduce inference cost by activating only a sparse subset of parameters per token. However, when these models exceed single-GPU memory, practitioners must distribute them across devices using tensor parallelism (TP), expert parallelism (EP), or a hybrid of both, and there is little empirical guidance on which strategy to choose. We present a controlled comparison of these strategies for MoE inference serving on NVIDIA A100 GPUs using vLLM. Across multiple models and workload settings, we find that EP achieves 85–102% of TP throughput depending on expert count, matching or exceeding TP for models with 60–64 experts, while naive TP+EP hybrid placement underperforms both alternatives. We also show that a diffusion-based MoE language model is substantially slower than its dense counterpart, highlighting the need for multi-GPU expert distribution in this emerging class. Overall, our results suggest that request concurrency has a larger impact on throughput than parallelism choice in many cases, and that EP is a strong default for large-expert MoE models.