统一了两种强化学习优化方法，揭示其本质是代理奖励最大化。
Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
- 通过逆向解析现有方法，发现优势塑造实质是优化代理奖励。
- 提出硬样本加权可视为奖励层面的正则化机制。
- 为新算法设计提供通用思路，适用于多种验证任务。
本文统一了强化学习中针对可验证奖励的Pass@K目标的两种不同策略：直接的REINFORCE类方法与优势塑造技术。我们证明二者本质相同。通过逆向分析现有优势塑造算法，发现其隐式地优化代理奖励。特别地，将实际应用中的‘硬样本加权’修改解释为奖励级正则化。反过来，从代理奖励目标出发，提出一个简单方法来推导现有及新优势塑造算法。该视角不仅深化了对Pass@K优化的理解，也为更广泛的RLVR策略梯度优化提供了新视角。
原文摘要 · Abstract (English)
This note reconciles two seemingly distinct approaches to policy gradient optimization for the Pass@K objective in reinforcement learning with verifiable rewards: (1) direct REINFORCE-style methods, and (2) advantage-shaping techniques that directly modify GRPO. We show that these are two sides of the same coin. By reverse-engineering existing advantage-shaping algorithms, we reveal that they implicitly optimize surrogate rewards. We specifically interpret practical "hard-example up-weighting" modifications to GRPO as reward-level regularization. Conversely, starting from surrogate reward objectives, we provide a simple recipe for deriving both existing and new advantage-shaping methods. This perspective provides a lens for RLVR policy gradient optimization beyond our original motivation of Pass@K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。