解决低资源语言数据不足问题,提出动态调整采样策略提升训练效果。
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets
- 先大力采样低资源语言加速收敛,再逐步减少采样防过拟合。
- 在随机梯度下降下,采样法比加权法梯度方差更小,收敛更快。
- 适合多语言模型训练,尤其对数据严重不均衡场景有效。
不同领域间的数据分布呈现长尾特征:少数领域数据丰富,多数领域面临数据稀缺。本文聚焦多语言场景,其中数据高度偏向高资源语言。现有两种常见应对策略为对低资源语言进行数据重采样(温度采样)和对低资源损失项进行加权(标量法)。尽管二者常被视为等价,但尚未严格验证。通过理论与实证分析,我们揭示了二者在全梯度下降下等价,但在随机梯度下降下因梯度方差差异而不同。温度采样具有更低的梯度估计方差,收敛更快,但过拟合风险更高。基于此,我们提出Cooldown策略:初期高强度采样低资源语言以加速收敛,随后逐步降低采样比例以抑制过拟合,实现两者优势结合。该方法在性能上媲美现有重加权技术,同时具备更高的计算效率。
原文摘要 · Abstract (English)
Data abundance across different domains exhibits a long-tailed distribution: few domains have abundant data, while most face data scarcity. Our work focuses on a multilingual setting, where available data is heavily skewed towards high-resource languages. Two common strategies to address this disparity are upsampling low-resource data (Temperature Sampling) and upweighting low-resource loss (Scalarization). These methods are often assumed to be equivalent, but this equivalence has not been rigorously established, prompting our investigation. Through theoretical and empirical analysis, we identify when these two methods are equivalent and when they diverge. We prove that they are equivalent under full gradient descent but differ under stochastic gradient descent due to differences in gradient variance. Specifically, Temperature Sampling exhibits lower variance in gradient estimation compared to Scalarization, leading to faster convergence but a higher risk of overfitting. Based on these insights, we propose Cooldown, a strategy that starts by heavily upsampling low-resource languages to accelerate convergence and gradually reduces the upsampling to prevent overfitting -- achieving the best of both worlds. Our method competes effectively with existing data re-weighting techniques while offering computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。