提出WAPO方法，提升语言模型强化学习的训练稳定性。
A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
- 基于梯度动态分析，构建预测更新影响的分类体系。
- 在数学推理与多跳问答任务中，稳定性和性能优于基线。
- 适合追求高效可靠强化学习训练的语言模型研究者。
基于可验证奖励的强化学习（RLVR）提升了语言模型的推理能力，但GRPO类优化仍易发生崩溃。本文通过分析逐标记的梯度动态，提出一个分类体系，用于预测更新对下一标记概率和熵的影响。该体系表明，稳定性取决于当前策略下优势符号与标记分布的联合作用。基于此，我们提出胜者优势策略优化（WAPO），一种仅对正优势生成结果进行在线裁剪的策略梯度目标。在数学推理与多跳问答基准上，WAPO显著提升训练稳定性，并在多个模型族中达到或超越基线表现。完整代码见 https://github.com/layer6ai-labs/wapo。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse. We analyse this instability through token-level gradient dynamics, deriving a taxonomy that predicts how updates affect next-token probabilities and entropy. The taxonomy shows that stability depends jointly on the advantage sign and token distribution under the current policy. Motivated by this finding, we propose Winner Advantage Policy Optimization (WAPO), a simple online clipped policy-gradient objective that updates only on positive-advantage completions. Across mathematical reasoning and multi-hop QA benchmarks, WAPO improves training stability and matches or outperforms baselines across multiple model families. Full code can be found at https://github.com/layer6ai-labs/wapo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。