arXiv:2606.08167cs.LGcs.AI2026-06被引 1

提出统一框架解释多领域数据混合的性能规律,能预测大尺度下最优训练组合。

Explaining Data Mixing Scaling Laws

  • 基于技能重叠与专长差异假设,构建容量竞争与噪声降低双机制模型
  • 在小规模数据上训练的模型可准确预测大规模未见场景下的最佳数据混合方案
  • 相比现有方法参数更少、误差更低,适合高效设计多领域训练策略

近期研究建立了用于预测多领域数据混合模型性能的经验缩放定律,但对其损失行为的理论理解仍为空白。本文提出一个统一框架,解释数据混合的内在机制。该方法将标准神经网络缩放定律(如Kaplan和Chinchilla)的理论视角扩展至多领域场景。基于各领域在基础技能上重叠、专业技能上分化这一分布假设,我们识别出两个决定模型在不同数据混合下损失的关键因素:容量竞争(有限模型容量导致各领域损失全局耦合)与噪声减少(最优权重向更难学习领域倾斜以最小化整体噪声)。实验表明,本框架在拟合损失景观时相对误差更低,且能识别出更高性能的训练混合方案。更重要的是,模型具备良好的外推能力,仅用小规模数据训练的参数即可有效预测大规模未见场景下的最优混合。此外,相比以往经验法则,本模型使用参数显著更少。代码已公开于https://github.com/meiqwq/Explaining-Data-Mixing-Scaling-Laws。

原文摘要 · Abstract (English)

Recent research has established empirical scaling laws to predict model performance on multi-domain data mixtures. However, a theoretical understanding of these model loss behaviors remains absent. In this work, we propose a unified framework to explain the underlying mechanics of data mixing. Our approach extends theoretical perspectives originally developed for standard neural scaling laws (e.g., Kaplan and Chinchilla) to the multi-domain setting. Based on the distributional assumption that domains overlap on fundamental skills while diverging on specialized skills, we identify two key factors that govern the domain losses of models trained on different data mixtures: \textit{Capacity Competition}, where the allocation of finite model capacity couples domain losses globally, and \textit{Noise Reduction}, where optimal weights shift toward harder-to-learn domains to minimize overall noise. Empirical evaluations show that our framework outperforms existing baselines by fitting the loss landscape with a lower Mean Relative Error and identifying higher-performing training mixtures. Most importantly, our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales using parameters fitted on smaller ones. In addition, our model achieves these results using significantly fewer parameters compared to previous empirical laws. Our code is available at https://github.com/meiqwq/Explaining-Data-Mixing-Scaling-Laws.

数据混合缩放定律模型预测机器学习理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。