提出PASS框架,解决大模型推理强化学习中的信号污染与粒度错配问题。
Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners

- 独立标准化过程、结果与格式信号流,避免通道污染
- 按值分块并块内广播奖励,解决粒度不匹配问题
- 将累计目标转为平均密度得分,防止长度膨胀陷阱
Group Relative Policy Optimization (GRPO) 是大模型推理强化学习的常用方法,通常通过学习的过程奖励模型(PRMs)或策略蒸馏的KL信号来增强稀疏的结果奖励。然而,在GRPO的组标准化优势上叠加步骤级信号时,会引发三类结构性缺陷:过程、结果与格式信号流在组标准化中相互污染;过程信号粒度与逻辑决策粒度不匹配;累积回报求和导致长度膨胀或探索截断。本文提出PASS(Process Advantage Signal Shaping)——一种轻量级中间件,置于任意标量步骤级过程信号与GRPO剪裁代理之间,分别通过优势融合、按值分块和分长度转换,逐项修复上述问题。在数学推理与多跳问答两个领域,针对学习式PRM与策略蒸馏KL信号(含广义变体),并在两种组标准化算子下验证,PASS在所有配置中均稳定提升pass@1性能。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is a default recipe for process-supervised reinforcement learning of LLM reasoners, and dense process supervision -- via learned process reward models (PRMs) or on-policy-distillation KL signals -- is a common way to densify its otherwise weak outcome reward. Layering such a step-level signal on top of GRPO's group-standardized advantage, however, exposes three structural pathologies: \emph{channel contamination} between the pooled process, outcome, and format streams at group standardization; \emph{resolution mismatch} between the granularity of the process signal and the granularity of the logical decisions being credited; and a \emph{cumulative trap} by which GRPO's return-to-go sum surfaces either length inflation or truncated exploration depending on the sign regime of the signal. We propose \textbf{PASS} (\emph{Process Advantage Signal Shaping}), a compact middleware that sits between any scalar step-level process signal and GRPO's clipped surrogate and addresses the three pathologies in turn: \emph{Advantage Fusion} standardizes the three streams independently within each group, \emph{Chunk-by-Value} derives value-homogeneous chunks from the signal itself and broadcasts credit within each chunk, and \emph{Divide-Length} converts the cumulative objective into an average-value-density score. We validate PASS across two domains and two process-signal paradigms -- a learned PRM on mathematical reasoning and an on-policy-distillation KL signal (with a generalized variant) on multi-hop question answering -- and under two group-standardization operators. In every regime PASS delivers a consistent pass@1 gain over the corresponding GRPO baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。