提出CPGD算法，让语言模型强化学习更稳定高效
CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models
- 用KL散度约束策略漂移，动态调节更新幅度
- 在多个任务上显著提升性能且避免训练崩溃
- 适合追求稳定训练的后训练阶段强化学习研究者
基于规则的强化学习（RL）已显著提升语言模型（LMs）的推理能力，但现有方法如GRPO、REINFORCE++和RLOO常因大步长策略更新和不当裁剪导致训练不稳定甚至崩溃。为此，我们提出一种新算法CPGD：通过基于KL散度的策略漂移约束，动态正则化策略更新，并对比率对数实施裁剪机制以防止过度更新。我们提供了理论支持，并通过实证分析证明其能有效缓解先前方法的不稳定性。此外，实验显示CPGD在保持训练稳定的同时显著提升性能。实现兼顾理论严谨性与实用性，为语言模型后训练阶段的强化学习提供可靠替代方案。代码已开源。
原文摘要 · Abstract (English)
Recent advances in rule-based reinforcement learning (RL) have significantly improved the reasoning capability of language models (LMs) with rule-based rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO -- often suffer from training instability, where large policy updates and improper clipping can lead to training collapse. To address this issue, we propose Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel algorithm designed to stabilize policy learning in LMs. CPGD introduces a policy drift constraint based on KL divergence to dynamically regularize policy updates, and leverages a clip mechanism on the logarithm of the ratio to prevent excessive policy updates. We provide theoretical justification for CPGD and demonstrate through empirical analysis that it mitigates the instability observed in prior approaches. Furthermore, we show that CPGD significantly improves performance while maintaining training stability. Our implementation balances theoretical rigor with practical usability, offering a robust alternative for RL in the post-training of LMs. We release our code at https://github.com/ModalMinds/MM-EUREKA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。