arXiv:2604.02288cs.LGcs.AI2026-04被引 51

提出统一方法,让模型快速改进又长期稳定。

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

论文配图:Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
图 1 · 摘自论文原文
  • 根据样本正确性动态分配优化策略:对错的用细粒度修正,对的用标准强化。
  • 在五项评测中平均超越基线3.4%~6.3%,且每步计算成本降低17.2%。
  • 适合追求高效与稳定微调的大模型训练者,尤其关注早期提升和资源效率。

基于可验证奖励的强化学习(RLVR)已成为大语言模型后训练的标准范式。尽管群体相对策略优化(GRPO)广泛应用,其粗粒度信用分配会均匀惩罚失败生成,缺乏针对具体偏差的词元级聚焦。自蒸馏策略优化(SDPO)通过更密集的逐逻辑层监督实现快速早期改进,但常在长周期训练中崩溃。我们发现其晚期不稳定性源于两个根本缺陷:对已正确样本进行自蒸馏引入优化歧义,且自教师信号可靠性逐步下降。为此,我们提出样本路由策略优化(SRPO),一种统一的在线策略框架:将正确样本路由至GRPO的奖励对齐强化,失败样本则路由至SDPO的目标逻辑层修正。SRPO还引入熵感知的动态加权机制,抑制高熵、不可靠的蒸馏目标,突出高置信度目标。在五个基准与两种模型规模下评估,SRPO兼具SDPO的快速早期提升与GRPO的长期稳定性。其在五项基准上平均性能超越基线,对Qwen3-8B的五项平均提升达GRPO的3.4%和SDPO的6.3%,同时实现适中响应长度,并将每步计算成本降低最多17.2%。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.

强化学习大模型微调策略优化样本路由

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。