通过语义重分组与梯度平衡，高效提升大模型训练效果
R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
- 基于语义相似性重新划分数据域，生成更细粒度训练分组
- 利用训练中梯度构建格拉姆矩阵，动态优化数据混合比例
- 仅需0.01%额外计算，性能媲美或超越现有顶尖方法
数据混合策略在降低语言模型训练成本方面表现优异，但存在两大缺陷：其一，依赖预设的数据领域（如数据源、任务类型），难以捕捉关键语义细节；其二，计算开销随领域数量呈不可控增长。为此，本文提出R&B框架，通过语义相似性重分组（Regroup）生成更细粒度的训练域，并利用训练过程中获取的领域梯度所诱导的格拉姆矩阵，高效优化数据组成（Balance）。该方法无需额外计算即可获得评估信息（如损失或梯度）。在标准正则条件下进行理论分析，揭示了R&B相比非自适应混合策略的有效性。实验验证其在五个不同数据集（涵盖自然语言、推理及多模态任务）上的有效性。仅需0.01%额外计算开销，R&B即达到或超过当前最优数据混合策略的性能。
原文摘要 · Abstract (English)
Data mixing strategies have successfully reduced the costs involved in training language models. While promising, such methods suffer from two flaws. First, they rely on predetermined data domains (e.g., data sources, task types), which may fail to capture critical semantic nuances, leaving performance on the table. Second, these methods scale with the number of domains in a computationally prohibitive way. We address these challenges via R&B, a framework that re-partitions training data based on semantic similarity (Regroup) to create finer-grained domains, and efficiently optimizes the data composition (Balance) by leveraging a Gram matrix induced by domain gradients obtained throughout training. Unlike prior works, it removes the need for additional compute to obtain evaluation information such as losses or gradients. We analyze this technique under standard regularity conditions and provide theoretical insights that justify R&B's effectiveness compared to non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness of R&B on five diverse datasets ranging from natural language to reasoning and multimodal tasks. With as little as 0.01% additional compute overhead, R&B matches or exceeds the performance of state-of-the-art data mixing strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。