用模型合并技术高效搜索大模型预训练数据比例,省时省力提升性能。
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
- 通过加权合并各数据集训练的模型,预测最优数据混合比例。
- 在22T token数据上实现更优基准表现,搜索成本降低超90%。
- 适合大规模模型训练团队,尤其关注数据配置优化的研究者。
确定有效的数据混合比例是大语言模型预训练的关键,需在通用能力与数学、编程等高难度任务表现间取得平衡。然而,现有方法或依赖不可靠的小规模代理实验,或需代价高昂的大规模探索。为此,我们提出DeMix框架,利用模型合并预测最优数据比例:先在候选数据集上大规模训练组件模型,再通过加权合并生成数据混合代理。该范式将搜索与训练解耦,无需额外训练即可评估无限采样混合,显著提升混合发现效率。大量实验表明,DeMix突破了充分性、准确性与效率的权衡,在更低搜索成本下获得更高基准性能。此外,我们发布了DeMix Corpora——一个包含22万亿标记的高质量预训练数据集,含经验证的数据混合比例,支持开放研究。代码与数据集已开源。
原文摘要 · Abstract (English)
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such as math and code. However, identifying an optimal mixture remains an open challenge, as existing approaches either rely on unreliable tiny-scale proxy experiments or require prohibitively expensive large-scale exploration. To address this, we propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios. Instead of training proxy models for every sampled mixture, DeMix trains component models on candidate datasets at scale and derives data mixture proxies via weighted model merging. This paradigm decouples search from training costs, enabling evaluation of unlimited sampled mixtures without extra training burden and thus facilitating better mixture discovery through more search trials. Extensive experiments demonstrate that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost. Additionally, we release the DeMix Corpora, a comprehensive 22T-token dataset comprising high-quality pre-training data with validated mixtures to facilitate open research. Our code and DeMix Corpora is available at https://github.com/Lucius-lsr/DeMix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。