Entropy-Guided Dynamic Expert Selection in Mixture-of-Experts Models

We present Adaptive-K routing, a method that dynamically selects the number of experts in Mixture-of-Experts (MoE) models based on routing entropy. Instead of using a fixed top-k experts per token, our approach uses fewer experts when the router is confident (low entropy) and more experts when uncertain (high entropy). VALIDATED RESULTS (January 2026):• Mixtral 8×7B: 31.0% compute reduction• Qwen-MoE: 32.4% compute reduction • OLMoE-1B-7B: 24.7% compute reduction• NVIDIA Nemotron 3 Nano: 33.3% compute reduction (128 experts, Mamba2-Transformer Hybrid) KEY FINDINGS:• Multiplicative composition: Adaptive-K + 4-bit quantization + speculative decoding = up to 96% total compute reduction• Quality preserved: