按阶段匹配数据难度，提升小模型推理能力
Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning
- 分阶段设计训练数据：SFT学新技能，RL巩固已有能力
- 难样本用桥接机制生成更易学的监督信号
- 失败案例转为诊断性修复数据，迭代优化
小语言模型（SLMs）推理能力的后训练通常采用SFT-then-RL流程，但现有工作很少关注各阶段应使用何种数据。我们认为，数据策略应与SFT和强化学习（RL）的不同角色对齐：SFT更适合学习尚未掌握的推理技能，而RL更适合巩固模型已部分具备的能力。基于此，我们提出一种难度感知的SFT-then-RL框架，将训练数据按阶段划分。在SFT阶段处理困难样本时，引入桥接（Bridge）机制，将教师生成的原始推理轨迹转化为更易学习的监督信号；在RL阶段仍有未解决的困难样本时，通过批评微调（Critique Fine-Tuning）将零奖励失败案例转化为诊断、修复与新推理轨迹的监督信号，用于下一阶段SFT。在两个SLM上对五个推理基准的实验表明，该方法持续优于代表性SFT、蒸馏和强化学习基线。结果强调了在SFT与RL阶段协调数据难度对有效提升小模型推理能力的重要性。
原文摘要 · Abstract (English)
Post-training Small Language Models (SLMs) for reasoning typically follows an SFT-then-RL pipeline, yet existing work rarely considers what data should be learned at each stage. We argue that data strategy should be aligned with the distinct roles of SFT and RL: SFT is better suited for acquiring not-yet-mastered reasoning skills, while RL is better suited for consolidating skills that the model can already partially access. Based on this principle, we propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets. For hard samples in the SFT stage, we introduce a Bridge mechanism that transforms raw teacher-generated reasoning traces into more learnable supervision for SLMs. For hard samples that remain unsolved during RL, we apply Critique Fine-Tuning by converting all-zero-reward failures into diagnostic, repair, and new reasoning trace supervision for the next SFT stage. Experiments on two SLMs across five reasoning benchmarks show that our method consistently improves over representative SFT, distillation, and RL baselines. Our results highlight the importance of coordinating data difficulty across SFT and RL for effective SLM reasoning post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。