用监督学习辅助强化学习，让大模型更高效地完成复杂推理。
SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
- 根据奖励信号动态切换强化学习与监督微调
- 在稀疏奖励下样本效率提升，泛化能力更强
- 适合有高质量推理数据的复杂任务场景
大型语言模型越来越多用于复杂推理任务，其中常存在高质量离线数据，如专家标注的答案和提炼的推理轨迹。然而，在奖励稀疏的环境中，强化学习难以采样到成功轨迹，导致学习效率低下。同时，标准的在线策略强化学习方法并未利用这些代表正确推理路径的离线数据。我们提出 SuperRL，一种统一训练框架，可自适应地在强化学习与监督微调（SFT）之间交替。当某实例的所有回放均未获得奖励时，SuperRL 转为在精选的离线数据上进行 SFT。在多个推理基准上的大量实验表明，SuperRL 在样本效率、泛化能力和稀疏奖励下的鲁棒性方面均优于基线强化学习方法。
原文摘要 · Abstract (English)
Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajectories, leading to inefficient learning. At the same time, these offline trajectories that represent correct reasoning paths are not utilized by standard on-policy reinforcement learning methods. We introduce SuperRL, a unified training framework that adaptively alternates between RL and SFT. Whenever every rollout for a given instance receives zero reward, indicating the absence of a learning signal, SuperRL falls back to SFT on the curated offline data. Extensive experiments across diverse reasoning benchmarks show that SuperRL surpasses vanilla RL by delivering higher sample efficiency, stronger generalization, and improved robustness under sparse rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。