动态调整训练数据组合，提升大模型预训练效果。
RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories
- 利用代理训练轨迹预测不同阶段最优数据混合比例。
- 在250亿token数据上，13项任务均优于现有方法。
- 仅需25%的计算资源即可超越传统静态混合方案。
数据混合选择对大语言模型预训练至关重要。现有方法如RegMix通过在小规模代理运行上拟合回归模型，选择单一静态混合比例。我们提出RegMix-D，是RegMix的简单扩展，实现动态混合。关键观察是：代理运行不仅产生终点损失，还生成完整的损失轨迹，可用于进一步优化数据混合。通过在这些轨迹上训练回归模型，可预测多个训练阶段的最优混合比例。RegMix-D支持两种部署模式：离线模式在目标训练前生成完整混合调度表；在线模式根据实际观测损失动态调整混合比例。在250亿token的Pile数据集上，使用10亿参数目标模型的实验表明，RegMix-D在13项下游任务中持续优于RegMix和DoReMi，同时保持代理效率：即使仅使用128个代理模型（仅为RegMix代理计算预算的25%），仍能超越其性能。
原文摘要 · Abstract (English)
Data mixture selection is critical for Large Language Model pretraining. Existing methods such as RegMix select a single static mixture by fitting a regression model on small-scale proxy runs. We propose RegMix-D, a simple extension of RegMix to dynamic mixing. Our key observation is that proxy runs produce not only endpoint losses, but also full loss trajectories, which can be used to further improve data mixture. By training regression model on these trajectories, we can predict optimal mixtures at multiple training stages. RegMix-D supports two deployment modes: an offline variant that generates a complete mixture schedule before target training, and an online variant that adapts the mixture during training using observed loss. Experiments on 25B tokens of the Pile dataset with a 1B parameter target model show that RegMix-D consistently improves over RegMix and DoReMi across 13 downstream tasks while remaining proxy-efficient: it surpasses RegMix even with only 128 proxy models (25% of RegMix's proxy compute budget).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。