提出新方法解决大模型强化学习中的梯度发散问题，提升推理稳定性与探索能力。
From $\log π$ to $π$: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
- 用概率梯度替代对数概率梯度作为优化基础，避免梯度权重发散。
- 在1.5B/7B/14B模型上均超越基线，在数学推理任务中表现更优。
- 适合追求稳定高效强化学习训练的大模型研究者使用。
基于可验证奖励的强化学习（RLVR）推动了大语言模型（LLM）推理能力的飞跃，但其优化过程仍易受干扰。标准算法如GRPO通过‘硬裁剪’保持稳定，却因丢弃信任区间外的梯度而抑制探索。近期的‘软裁剪’虽试图恢复这些梯度，但依赖∇θlog πθ会导致概率趋近零时梯度权重发散，造成训练不稳定。本文重新审视这一范式，将∇θπθ确立为更优的优化原语。为此提出解耦梯度策略优化（DGPO），基于重要性采样比设计非对称连续衰减机制，对边界词元施加解耦衰减，平衡稳定性与持续探索。在DeepSeek-R1-Distill-Qwen系列模型（1.5B/7B/14B）上的大量实验表明，DGPO在多个数学基准测试中持续优于强基线，为RLVR提供了鲁棒且可扩展的解决方案。代码已开源：https://github.com/FlyTune/DGPO-RL。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed a leap in Large Language Model (LLM) reasoning, yet its optimization dynamics remain fragile. Standard algorithms like GRPO enforce stability via "hard clipping", which inadvertently stifles exploration by discarding gradients of tokens outside the trust region. While recent "soft clipping" methods attempt to recover these gradients, they suffer from a critical challenge: relying on log-probability gradient ($\nabla_θ\log π_θ$) yields divergent weights as probabilities vanish, destabilizing LLM training. We rethink this convention by establishing probability gradient ($\nabla_θπ_θ$) as the superior optimization primitive. Accordingly, we propose Decoupled Gradient Policy Optimization (DGPO), which employs a decoupled decay mechanism based on importance sampling ratios. By applying asymmetric, continuous decay to boundary tokens, DGPO resolves the conflict between stability and sustained exploration. Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR. Our code and implementation are available at: https://github.com/FlyTune/DGPO-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。