用在线数据优化指令数据分布,让大模型学得更好。
DRIFT: Refining Instruction Data via On-Policy Data Attribution

- 基于模型自生成结果做数据重要性评估,更贴近实际训练场景。
- 在70亿参数模型上,显著提升指令和推理能力上限。
- 适合想突破现有数据集性能瓶颈的研究者与工程师。
监督微调(SFT)中优化训练数据分布对大语言模型能力至关重要。现有数据筛选方法虽能加速训练,却难以提升模型能力上限。问题已从‘找更小但有效的数据子集’转变为‘精炼数据分布以包含最能提升模型的样本’。为此,本文探索使用影响函数(Influence Functions, IF)进行实例级数据归因。发现标准IF因离线验证目标导致距离偏差,且严重偏向梯度范数。提出DRIFT(基于在线策略影响函数的数据精炼),不依赖外部参考数据,而是利用模型自身生成的输出作为验证目标,有效缩小参数距离差距,符合IF的局部邻域假设。同时引入基于轨迹正确性的符号加权,并缓解梯度操纵问题,使少量验证查询即可可靠地为全数据集分配影响力分数。在70亿参数的指令与推理模型上实验显示,DRIFT持续提升性能上限,优于现有数据筛选基线。
原文摘要 · Abstract (English)
Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs). While existing data curation methods excel at accelerating training under constrained budgets, they are less suited to elevating the capability upper bound. The challenge here is no longer to identify a smaller subset that preserves performance, but to refine the data distribution toward instances most capable of improving the final model. To address this problem, we explore instance-level data attribution using Influence Functions (IF). We identify that standard IF formulations struggle in this setting due to two structural limitations: a proximity gap caused by off-policy validation targets, and a severe bias towards gradient norm. We propose DRIFT (Data Refinement via On-Policy Influence Functions for Supervised Fine-Tuning). Instead of relying on external reference data, DRIFT utilizes the model's on-policy rollouts as validation targets, which empirically minimizes the parameter proximity gap and better aligns with the local neighborhood assumption of IF. It further applies signed weighting based on trajectory correctness and debiases influence scores against the gradient hacking issue, allowing a small set of validation queries to act as reliable anchors for attributing the full dataset. Experiments on 7B-parameter instruction and reasoning models show that DRIFT consistently raises the performance ceiling on both, outperforming existing data curation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。