Large-Scale Diverse Synthesis for Mid-Training

Mid-training has become critical for enhancing the knowledge and reasoning ability of large language models (LLMs), especially through the utilization of large-scale synthetic data.However, existing data synthesis methods often generate simplistic and homogeneous QA pairs, with limited scale and diversity.To address this, we propose BoostQA, a novel framework designed to synthesize large-scale, diverse, and high-quality QA data for midtraining.BoostQA introduces model probes during mid-training for the first time and implements STEM-focused multi-grade synthesis to boost data diversity as well as high-difficulty synthesis to alleviate difficulty degradation, followed by answer refinement to further improve quality.Extensive experiments by mid-training Llama-3 8B demonstrate that using only 20Btoken BoostQA data achieves a significant average improvement of 12.74% on MMLU and CMMLU over the pre-training baseline.After mid-training on 500B tokens, including 100Btoken BoostQA data, our model achieves SOTA average results across benchmarks among mainstream models of comparable size.BoostQA also demonstrates robust scalability, with performance consistently improving as model size, data volume, and initial FLOPs scale. 1 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 MMLU 73.6 65.5 70.7 74.1 72.0 76.2 G L M 4 -9 B -B a s e L l a m a 3 .1 -8 B -B a s e Y i -1 .5 -9 B -B a s e Q w e n 2 .5 -7 B -B a s e G e m m a 2 -9 B -B a s e O u r s 0 20 40 60 80 CMMLU 75.0 53.1 75.0 82.8 59.1 80.