让AI模型在推理时更灵活,根据自身表现动态调整学习策略。
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning

- 根据模型自身推理表现动态调整专家指导强度
- 在模型自信时允许自主生成解法,提升泛化能力
- 适合需要可靠推理的数学与代码任务
监督微调(SFT)通过模仿离线专家示范来适应大语言模型的推理任务,通常将单一专家轨迹视为目标行为。但推理并非简单路径模仿:机械遵循一个示范可能过拟合表层形式,抑制模型自身的推理分布。我们提出滚动自适应监督微调(RASFT),一种政策感知的SFT框架,根据从验证的在线回溯中估计的问题可解性校准专家监督。对于每个问题,当当前策略表现困难时,RASFT增强专家引导;当模型已展现可靠推理行为时,则放松严格模仿并融入正确的自我生成轨迹。为保留有用的推理先验,RASFT进一步引入冻结参考模型与当前策略之间的裁剪逆比例,以限制过度策略漂移。在六个数学推理基准和两个代码推理基准上,多个模型的实验表明,RASFT在整体性能上优于SFT、SFT变体及代表性强化学习方法。代码已公开于https://github.com/zjd1sq/RASFT。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is a prevailing method for adapting large language models to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However, reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model's own reasoning distribution. We propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware SFT framework that calibrates expert supervision according to problem-level solvability estimated from verified on-policy rollouts. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self-generated trajectories when the model already exhibits reliable reasoning behavior. To preserve useful reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift. Experiments across multiple models on six mathematical reasoning benchmarks and two code reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods. The code is available at https://github.com/zjd1sq/RASFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。