小模型也能做多步推理，靠的是逐步引导的强化学习新框架
Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- 让模型先生成内部推理过程，再执行每一步动作
- 即使所有尝试都错，仍能通过相似度提供平滑奖励信号
- 适合需要逻辑推理的小型开源模型，也适用于编程等任务
大型语言模型在需要多步推理的问题上表现不佳。对于小型开源模型，基于可验证奖励的强化学习（RLVR）在正确解极难采样时失效，而监督微调（SFT）则因逐标记模仿导致过拟合。为此，我们提出监督强化学习（SRL），将问题求解重构为生成一系列逻辑“动作”的过程。SRL 训练模型在每一步动作前生成内部推理对话，并基于模型动作与来自 SFT 数据集的专家动作之间的相似性，以逐步方式提供平滑奖励。这种监督机制即使在所有回放均错误时仍能提供丰富学习信号，同时鼓励受专家示范引导的灵活推理。结果表明，SRL 使小型模型能够学习此前无法通过 SFT 或 RLVR 掌握的挑战性问题。此外，先用 SRL 初始化，再用 RLVR 精炼，可获得最佳整体性能。SRL 在推理基准之外也有效泛化至代理式软件工程任务，展现出作为推理导向大模型训练框架的稳健性与通用性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。