让奖励传递更贴近模型计算路径，提升大语言模型强化学习效果
Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning
- 根据模型内部计算动态调整奖励传播方式，而非固定规则
- 在5个Qwen3-4B实验中达61.4%准确率，优于基线53.8%
- 适用于需要稳定强化学习训练的大模型研究者
大语言模型强化学习中的信用分配可分为三部分：成功证据、将证据转化为词元级优势的传输算子，以及将优势转化为策略更新的几何结构。现有工作已显著改进证据获取、采样和更新机制，但传输算子通常与架构无关。固定折扣GAE采用沿词元时间的静态几何核；组相对方法将结果统计广播至整个响应。两者均未反映Transformer策略自身轨迹特定的计算过程。本文提出计算条件信用传输（CCT）框架，通过行为策略内部计算的分离统计量参数化因果核，实现下游价值的路径依赖传播。具体算法CompPO将注意力集中度映射为有界词元保留门，用于单步自举与路径依赖的广义优势迹（Comp-GAE），并设计可复用演员隐藏状态与路由信息的传输对齐评论家（TAC），无需第二个体型相同的Transformer。任务奖励与裁剪PPO目标保持不变；常数门退化为固定系数GAE。在五个Qwen3-4B种子下，CompPO达到61.4%最终保留集准确率（95% CI [60.8,62.0]），优于调优后的GRPO（53.8% [52.9,54.7]）。仅使用Comp-GAE或仅使用固定门的TAC均无法达到完整模型效果（交互提升+2.4 [1.9,2.9]）。随机与位置控制验证轨迹特异性对齐；在12次PPO网格测试中，CompPO稳定运行10次，而基线仅3次。冻结评估在Qwen3-4B和Llama-3.1-8B-Instruct上分别优于GRPO 4.3和3.9个贪婪通过@1宏指标点。
原文摘要 · Abstract (English)
Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport operator that converts this evidence into token-level advantages, and an update geometry that turns advantages into policy changes. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture-agnostic. Fixed-discount GAE applies a stationary geometric kernel along token time; group-relative methods broadcast an outcome statistic across an entire response. Neither operator represents the trajectory-specific computation used by the Transformer policy itself. We introduce computation-conditioned credit transport (CCT), a general framework in which a detached statistic of the behavior policy's internal computation parameterizes the causal kernel that transports downstream value through a rollout. Our concrete algorithm, CompPO, maps native attention concentration to a bounded per-token retention gate, uses the gate in both the one-step bootstrap and a path-dependent generalized-advantage trace (Comp-GAE), and co-designs a transport-aligned critic (TAC) that reuses the actor's hidden states and routing information without a second same-scale Transformer. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches 61.4% final held-out accuracy (95% CI [60.8,62.0]) versus 53.8% [52.9,54.7] for tuned GRPO. Neither Comp-GAE with a standard critic (55.2%) nor TAC with a fixed gate (56.4%) matches the full model (interaction +2.4 [1.9,2.9]). Shuffle and position controls confirm trajectory-specific alignment; CompPO is stable in 10/12 PPO-grid runs versus 3/12. Frozen evaluation improves over GRPO by 4.3 and 3.9 greedy pass@1 macro points on Qwen3-4B and Llama-3.1-8B-Instruct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。