用熵比裁剪稳定强化学习,解决策略更新过冲问题。
Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
- 以当前与历史策略的熵比为全局指标,量化探索变化
- 双向裁剪熵比,抑制未采样动作的概率偏移
- 在多个基准上提升稳定性与性能,适合语言模型微调
大型语言模型后训练依赖强化学习提升能力与对齐质量。然而,离策略训练引入分布偏移,常使策略超出信任区域,导致策略熵波动和梯度不稳。尽管PPO-Clip通过重要性裁剪缓解此问题,仍忽略动作的全局分布变化。为此,我们提出使用当前与先前策略的熵比作为新全局度量,有效量化策略探索的相对变化。基于此,我们引入熵比裁剪(ERC)机制,在熵比上施加双向约束,从全局分布层面稳定策略更新,并弥补PPO-Clip无法调控未采样动作概率偏移的缺陷。我们将ERC集成至DAPO与GPPO算法中。多基准实验表明,ERC持续提升性能。
原文摘要 · Abstract (English)
Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an \textbf{Entropy Ratio Clipping} (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。