通过追踪梯度轨迹，用极少数据实现模型性能最大化
Influential Language Data Selection via Gradient Trajectory Pursuit
- 联合选择数据点，自动去重，避免传统方法的重复采样
- 仅需0.5%数据即可达到完整训练效果，显著降低数据量
- 适合大规模语言模型训练中的高效数据筛选场景
构建高性能大语言模型的核心在于数据集的精心筛选（Touvron等, 2023；Achiam等, 2023；Team等, 2024）。梯度影响分数（Pruthi等, 2020；Xia等, 2024）被证明与模型性能相关，常用于数据选择。然而，现有方法依赖单一样本排序或低效匹配过程，导致性能不佳或难以扩展。本文提出梯度轨迹追逐（Gradient Trajectory Pursuit, GTP），通过在L0范数正则化目标下联合选择数据点，实现梯度轨迹追踪。该算法具有：（1）联合选择而非独立选Top-k，自动去重；（2）高效压缩采样流程，可借助分布式框架进一步加速。实验表明，在域内与目标域数据选择基准上，GTP持续优于Top-k选择和竞争性算法，例如仅使用0.5%的数据即可在指令微调任务中达到全性能。
原文摘要 · Abstract (English)
Curating a desirable dataset for training has been the core of building highly capable large language models (Touvron et al., 2023; Achiam et al., 2023; Team et al.,2024). Gradient influence scores (Pruthi et al., 2020; Xia et al., 2024) are shown to be correlated with model performance and are commonly used as the criterion for data selection. However, existing methods are built upon either individual sample rankings or inefficient matching process, leading to suboptimal performance or scaling up issues.In this paper, we propose Gradient Trajectory Pursuit (GTP), an algorithm that performs pursuit of gradient trajectories via jointly selecting data points under an L0-norm regularized objective. The proposed algorithm highlights: (1) joint selection instead of independent top-k selection, which automatically de-duplicates samples; (2) higher efficiency with compressive sampling processes, which can be further sped up using a distributed framework. In the experiments, we demonstrate the algorithm in both in-domain and target-domain selection benchmarks and show that it outperforms top-k selection and competitive algorithms consistently, for example, our algorithm chooses as low as 0.5% data to achieve full performance on the targeted instruction tuning tasks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。