根据领域重要性动态采样训练数据，提升大模型性能。
DIDS: Domain Impact-aware Data Sampling for Large Language Model Training
- 用梯度聚类保证同一领域数据学习一致性。
- 基于FIM的指标量化领域更新对下游任务影响，提升3.4%平均表现。
- 适合需要高效多领域训练的大模型研究者使用。
大型语言模型通常在多领域数据集上训练，不同领域的相对重要性显著影响下游任务表现。现有领域采样策略难以兼顾领域内一致性与领域影响的准确评估。本文提出领域影响感知的数据采样方法（DIDS）。为确保领域内一致性，采用梯度聚类算法，通过代理语言模型和降维技术降低计算开销；为精确测量领域影响，设计基于费雪信息矩阵（FIM）的度量指标，理论证明其能有效反映领域参数更新对下游输出分布的影响。此外，DIDS结合FIM评估结果与损失学习轨迹，考虑边际效益递减，优化采样比例。大量实验表明，DIDS在保持相近训练效率的同时，平均性能提升3.4%。代码已公开于 https://github.com/shiweijiezero/DIDS。
原文摘要 · Abstract (English)
Large language models (LLMs) are commonly trained on multi-domain datasets, where domain sampling strategies significantly impact model performance due to varying domain importance across downstream tasks. Existing approaches for optimizing domain-level sampling strategies struggle with maintaining intra-domain consistency and accurately measuring domain impact. In this paper, we present Domain Impact-aware Data Sampling (DIDS). To ensure intra-domain consistency, a gradient clustering algorithm is proposed to group training data based on their learning effects, where a proxy language model and dimensionality reduction are employed to reduce computational overhead. To accurately measure domain impact, we develop a Fisher Information Matrix (FIM) guided metric that quantifies how domain-specific parameter updates affect the model's output distributions on downstream tasks, with theoretical guarantees. Furthermore, to determine optimal sampling ratios, DIDS combines both the FIM-guided domain impact assessment and loss learning trajectories that indicate domain-specific potential, while accounting for diminishing marginal returns. Extensive experiments demonstrate that DIDS achieves 3.4% higher average performance while maintaining comparable training efficiency. The code is available at https://github.com/shiweijiezero/DIDS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。