提出新方法，高效优化大模型训练数据组合，省一半成本且性能提升3%。
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
- 基于模型容量设计混合规律，精准建模数据组合与模型大小关系。
- 仅需一半计算成本即可完成数据混合优化，55B模型性能提升3%。
- 适合追求高效训练的大模型研发团队，尤其适用于专家混合模型。
数据混合指不同数据源在训练大语言模型时的组合方式，选择有效混合策略对下游性能至关重要。现有方法要么直接在目标模型上进行高成本搜索，要么依赖混合缩放定律，难以外推至大模型规模。本文提出一种计算高效的混合缩放管道：首先引入CAMEL，一种考虑模型容量的混合规律，能建模验证损失与模型规模及混合策略间的非线性关系；其次提出损失到基准预测法则，从验证损失估算基准准确率，实现目标模型端到端性能预测。接着研究如何在固定算力预算下分配不同模型规模的资源以拟合该规律并降低预测误差。最后将方法应用于参数量达7B-A150M的专家混合模型，拟合规律后外推至55B-A1.2B目标模型，并验证其最优混合策略。相比以往方法，混合优化成本降低50%，下游基准性能最高提升3%。
原文摘要 · Abstract (English)
A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target model. Compared to prior methods, we reduce mixture optimization costs by 50\% and improves downstream benchmark performance by up to 3\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。