PRPO让大模型推理更准，用过程奖励提升中间步骤指导性
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
- 分段处理推理过程，将过程奖励转化为逐标记优势
- 在MATH500上用8次采样使准确率从61.2%提至64.4%
- 无需价值网络，适合资源有限的推理优化场景
大语言模型在多步推理任务中常面临奖励信号稀疏问题。无价值网络方法如GRPO为所有标记分配单一归一化结果奖励，难以指导中间推理。尽管过程奖励模型（PRMs）提供密集反馈，但单独使用易导致早期低奖励标记引发输出截断，造成过早崩溃。本文提出过程相对策略优化（PRPO），在无价值网络框架下结合结果可靠性与过程级指导。PRPO基于语义线索对推理序列进行分段，将PRM得分归一化为逐标记优势，并通过位置-参数变换使其分布与结果优势对齐。在MATH500数据集上，仅使用8次采样且无需价值网络，PRPO将Qwen2.5-Math-1.5B的准确率从61.2%提升至64.4%，验证了其在无监督框架下的高效细粒度信用分配能力。代码已开源：https://github.com/SchumiDing/srpocode
原文摘要 · Abstract (English)
Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。