DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections

As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge.To address this, we propose DY-NAMIXSFT, a dynamic and automated method for instruction-tuning dataset mixture optimization.We formulate the problem as a multiarmed bandit setup and introduce a Priorscaled Boltzmann Exploration that softly anchors the updated sampling distribution to the original dataset proportions, thereby preserving the inherent diversity and coverage of the collection.Sampling probabilities are updated using a lightweight 1-Step Look-ahead Reward, reflecting how much the dataset contributes to improving the model's performance at its current state.We demonstrate that DYNAMIXSFT effectively optimizes the TÜLU-2-mixture and TÜLU-3-mixture collections across 10 benchmarks, while introducing minimal computational overhead over naive sampling.Furthermore, we provide a comprehensive analysis and visualizations to offer deeper insights into the adaptive dynamics of our method.