Scaling Text-Based Semantic Communication Systems With Heterogeneous Mixture of Experts

To address scalability in semantic communication (SC), we propose a novel heterogeneous Mixture-of-Experts (HMoE)-based SC (HMoE-SC) design, particularly for text-based systems. Our model employs a heterogeneous expert pool and a channel-aware gating mechanism that dynamically selects experts to efficiently deal with the input token complexity and enhance the robustness against adverse channel conditions. This enables efficient scaling of model capacity, activating lightweight experts in favorable scenarios and larger ones when needed, thereby balancing task performance and computational efficiency. Experimental results on a semantic similarity classification task using the Quora Question Pairs dataset confirm our HMoE-SC design improves semantic accuracy while efficiently reducing inference cost compared to the state-of-the-art models.