提出动态优化框架,让大模型推理更稳定且更智能。
Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

- 用分组对齐损失降低强化学习梯度波动
- 多教师蒸馏纠正监督微调的偏差,提升准确性
- 根据奖励反馈自适应切换稳定与探索模式
大语言模型后训练范式中,监督微调(SFT)虽稳定但易过拟合,强化学习(RL)可探索但梯度方差高。本文提出DYPO(动态策略优化)框架,通过三组件协同:(1) 分组对齐损失(GAL)利用内在群体动态显著降低RL梯度方差;(2) 多教师蒸馏机制通过多样化推理路径修正SFT的拟合偏差;(3) 动态利用-探索门控机制依据奖励反馈自适应权衡稳定与探索。理论分析表明,DYPO线性降低拟合偏差并最小化整体方差。实验显示,相比传统串行流程,DYPO在复杂推理基准上平均提升4.8%,在分布外任务上提升13.3%。代码已开源。
原文摘要 · Abstract (English)
Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose \textbf{DYPO} (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a \textit{Group Alignment Loss (GAL)} that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a \textit{Multi-Teacher Distillation} mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a \textit{Dynamic Exploitation-Exploration Gating} mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8\% on complex reasoning benchmarks and 13.3\% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。