Anchor-Stabilized Granular Mixture of Experts for Low-Resource Language Model Training

This preprint introduces Anchor-Stabilized Granular Mixture of Experts (ASG-MoE), a hybrid Mixture-of-Experts architecture designed for training language models under low-resource constraints. Small dense transformers trained on consumer hardware frequently suffer from repetition collapse, unstable optimization, and failure to acquire syntactic coherence. Conversely, naïve sparse MoE architectures at small scale often collapse due to routing instability. ASG-MoE addresses this failure mode by introducing a permanently active shared Anchor expert responsible for syntactic stability, alongside multiple lightweight Specialist experts selected via sparse routing. This design preserves gradient flow during sparsity-induced instability while enabling emergent specialization. The model was trained entirely on consumer hardware (Apple M4, 16GB unified memory) using the MLX framework. Experimental results demonstrate stable convergence, recovery from sparse routing shock, balanced expert utilization, and clear qualitative separation between syntactic competence and semantic specialization. This work contributes an architectural inductive bias for stabilizing small-scale MoE models and demonstrates the feasibility of server-style routing architectures on edge-class hardware. Associated resources: Hugging Face model: https://huggingface.co/dipeshmajithia/Mirror-80M-MoE This artifact is intended as a research preprint and reference design for low-resource language model training.