通过过滤已掌握内容,让模型更专注学习薄弱环节,提升强化训练效果。
TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

- 训练时过滤已掌握的样本,聚焦数据分布尾部的低频内容
- 在数学与编程任务上,pass@16性能最高提升17%,后续强化训练增益达4%
- 提供轻量诊断工具,帮助判断何时使用该方法最有效
强化学习后训练推动现代AI系统产生推理与代理能力,但现有研究表明,其效果最佳时依赖于已有较强基础的模型。我们质疑现有流程是否真正为强化学习准备了最优模型。基于先前工作指出覆盖度与pass@K可预测后强化学习表现,本文提出简单的监督微调改进方法TailSFT:在训练中过滤已充分学习的序列,使学习集中于数据分布尾部的未充分建模区域。通过受控实验与理论分析验证了设计选择,尤其是具体过滤标准。在OLMo-3 7B上,TailSFT常显著提升数学与编程评估中的pass@16表现,最高绝对提升17%,且计算开销极小。这些高覆盖度检查点在后续GRPO训练中带来最多4%的pass@1绝对提升,证明其作为强化学习初始化更具优势。此外,我们引入一种轻量级诊断工具,识别出最可能受益于TailSFT的场景。总体而言,结果支持一种分阶段、有原则的模型开发范式,即中间检查点应以支持后续训练的能力来评估。
原文摘要 · Abstract (English)
Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。