通过跳连结构分离推理阶段，提升强化学习中的数学与代码生成性能。
Skip-Connected Policy Optimization for Implicit Advantage
- 将推理分为上游密集奖励与下游组相对优化，用跳连融合两者
- 在数学与代码任务上分别超越最强基线3.91%和6.17%
- 即使最终正确率相同，中间步骤质量也更高，适合高阶推理任务
组相对策略优化（GRPO）在强化学习求解视觉推理任务中表现优异，依赖结果奖励。尽管细粒度密集奖励理论上可提升性能，但在实际采样预算下，蒙特卡洛估计对早期推理标记产生高方差且符号不一致的优势，反而导致性能低于仅使用结果奖励的GRPO。为此提出跳连优化（SKPO），将推理过程分解为上游与下游阶段：上游通过单流蒙特卡洛采样获得密集奖励并进行优化；下游保持组相对优化，并通过跳连将上游段与原始问题拼接，使模型能利用有益的上游推理，同时保留绕过错误推理直接访问问题的能力。实验显示，在Qwen2.5-Math-7B与Llama-3.2-3B上，分别在数学基准与跨领域任务（包括通用推理与代码生成）中取得3.91%与6.17%的相对提升。进一步分析揭示隐式优势：即便最终正确率匹配，SKPO生成的轨迹在中间步骤质量更高。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。