用解耦优势归一化，让模型既答对题又理得清。
PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization
- 将结果与过程奖励分开归一化，避免评分干扰训练
- 在奥数数据集上达到51.3%准确率，优于原有方法
- 适合需要高质量推理的复杂任务训练
我们提出过程感知策略优化（PAPO），通过解耦优势归一化，将过程级评估融入组相对策略优化（GRPO），以解决现有奖励设计的两个缺陷。结果奖励模型（ORM）仅评估最终答案正确性，忽略推理质量，且当各组表现趋同后优势信号逐渐消失；过程奖励模型（PRM）提供更丰富监督，但直接使用其分数会导致奖励劫持，模型通过冗长表达虚增得分而准确率下降。PAPO通过组合来自ORM的全局归一化结果优势项Aout，和来自基于评分量表的PRM的仅在正确回答间归一化的过程优势项Aproc来解决上述问题。该解耦设计确保Aout锚定训练于正确性，而Aproc在不扭曲结果信号的前提下区分推理质量。多尺度模型在六个基准上的实验表明，PAPO持续优于ORM，尤其在奥数基准上达到51.3%准确率，相较ORM的46.3%显著提升，且在ORM趋于饱和甚至下滑时仍保持增长。
原文摘要 · Abstract (English)
We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, treating all correct responses identically regardless of reasoning quality, and gradually lose the advantage signal as groups become uniformly correct. Process reward models (PRM) offer richer supervision, but directly using PRM scores causes reward hacking, where models exploit verbosity to inflate scores while accuracy collapses. PAPO resolves both by composing the advantage from an outcome component Aout, derived from ORM and normalized over all responses, and a process component Aproc, derived from a rubric-based PRM and normalized exclusively among correct responses. This decoupled design ensures that Aout anchors training on correctness while Aproc differentiates reasoning quality without distorting the outcome signal. Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM, reaching 51.3% vs.\ 46.3% on OlympiadBench while continuing to improve as ORM plateaus and declines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。