调节多源数据训练权重，提升模型泛化能力
Sampling and Loss Weights in Multi-Domain Training
- 区分采样权重与损失权重，分别控制数据贡献和梯度更新
- 理论证明两类权重可降低梯度方差并缩小泛化差距
- 适用于多领域数据训练的模型优化，尤其在数据异构时
大型深度神经网络训练需要海量数据，通常从多个异构数据源（如 Wikipedia、GitHub）收集。不同数据源在质量与信息多样性上差异显著，如何合理分配各源权重成为关键问题。现有方法多通过启发式或近似方式设定采样权重。本文首次系统研究采样权重（控制每批数据中各域贡献）与损失权重（缩放各域损失）的双重作用。基于线性回归的严格分析表明，两类权重具有互补性：一方面可降低随机梯度下降（SGD）等迭代算法的梯度估计方差；另一方面能通过缩小泛化差距提升模型泛化性能。研究提供理论与实证支持，并探讨两者联合动态，揭示其协同捕捉各数据域贡献的机制。
原文摘要 · Abstract (English)
In the training of large deep neural networks, there is a need for vast amounts of training data. To meet this need, data is collected from multiple domains, such as Wikipedia and GitHub. These domains are heterogeneous in both data quality and the diversity of information they provide. This raises the question of how much we should rely on each domain. Several methods have attempted to address this issue by assigning sampling weights to each data domain using heuristics or approximations. As a first step toward a deeper understanding of the role of data mixing, this work revisits the problem by studying two kinds of weights: sampling weights, which control how much each domain contributes in a batch, and loss weights, which scale the loss from each domain during training. Through a rigorous study of linear regression, we show that these two weights play complementary roles. First, they can reduce the variance of gradient estimates in iterative methods such as stochastic gradient descent (SGD). Second, they can improve generalization performance by reducing the generalization gap. We provide both theoretical and empirical support for these claims. We further study the joint dynamics of sampling weights and loss weights, examining how they can be combined to capture both contributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。