arXiv:2506.17211cs.LG2025-06NeurIPS被引 57

用专家提示引导小模型推理,让训练更快更稳。

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning

  • 小模型卡在难任务时,用专家短提示辅助完成推理路径。
  • 只需不到40%真实答案,训练速度提升3倍,效果更优。
  • 适合资源有限但需强推理的小模型研究者使用。

小语言模型(SLMs)在缺乏高质量推理轨迹时难以学习复杂推理行为。标准训练流程为监督微调(SFT)后接强化学习(如GRPO),但在专家轨迹过难或模型初始成功率极低时可能完全失效。本文提出BREAD:一种通过部分专家引导与分支回放统一SFT与RL阶段的GRPO变体。当自生成轨迹失败时,BREAD自动插入短专家前缀,使小模型能完成后续推理,确保每次更新都有成功轨迹。该机制既增强奖励信号密度,又形成自然学习阶梯。实验表明,BREAD仅需少于40%的真实轨迹,持续优于标准GRPO,训练提速约3倍。更重要的是,它可解决原策略无法处理的问题,验证了分支回放与专家引导对提升小模型推理能力的关键作用。

原文摘要 · Abstract (English)

Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines a supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by a reinforcement learning (RL)stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RL stages via partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3 times. Importantly, we demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance can substantially boost SLM reasoning.

小模型推理强化学习专家引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。