让大模型的监督微调与强化学习协同进化,提升推理能力。
Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
- SFT通过选择性知识迁移,主动辅助强化学习优化奖励
- 在五个数学推理基准上平均提升超3分,训练更稳定
- 适合需要强推理能力且对噪声鲁棒的场景
监督微调(SFT)和可验证奖励强化学习(RLVR)是提升大语言模型推理能力的两种主流后训练方法。现有方法尝试将两者在单阶段中耦合,通过重加权或调度目标实现。但这种耦合可能适得其反,因为监督更新并非始终有利于奖励优化。为此,我们提出BRIDGE——一种可扩展框架,使SFT通过轻量级低秩适配器(LoRA)有选择地传递能提升奖励优化的知识。具体而言,每轮元训练交替执行:基础模型融合SFT与RL梯度;同时更新LoRA适配器,以最大化合作增益信号(即联合SFT-RL训练的奖励相对于仅RL基线的提升)。在五个数学推理基准上,BRIDGE持续优于两阶段冷启动、简单混合及代表性单阶段集成方法,平均绝对提升超过3分,训练动态更稳定。此外,该方法还可扩展至逻辑推理,并在无需额外训练的情况下泛化到代码和科学任务,且在噪声奖励下仍保持鲁棒。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) are two widely used post-training paradigms for improving the reasoning ability of large language models (LLMs). Recent methods attempt to integrate SFT and RLVR in a single stage by reweighting or scheduling their objectives. However, such coupling can be counterproductive because supervised updates are not uniformly beneficial for reward optimization. To address this, we propose BRIDGE, a scalable framework in which SFT learns to supervise RL by selectively transferring knowledge that improves reward optimization. Specifically, BRIDGE alternates two updates at each meta-training step: a base-model update that fuses the SFT and RL gradients, and an update to a lightweight low-rank adapter (LoRA) that coordinates the two objectives by maximizing a cooperative-gain signal, defined as the reward of joint SFT-RL training over an RL-only baseline. Across five mathematical reasoning benchmarks, BRIDGE consistently outperforms two-stage cold start, naive mixing, and representative single-stage integration baselines, yielding over three points average absolute improvement and more stable training dynamics. We further show that BRIDGE extends to logical reasoning and generalizes out-of-distribution to code and science without additional training, while staying robust under noisy rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。