改进强化学习中token的信用分配，提升推理任务表现
ACPO: Asymmetric Credit Policy Optimization via Mode-Local Entropy Surrogate
- 用顶词概率补数代替全局熵，实现局部模式感知的信用评估
- 在AIME 2025和HumanEval Pro上超越现有方法，数学与编码任务均显著提升
- 特别适合需要精细决策分析的复杂推理场景
结果监督的强化学习可扩展至可验证的推理任务，但轨迹级奖励对所有采样token赋予相同结果信号，忽略了它们在推理过程中的不平等贡献。熵能自然反映模型决策状态，但用于token级信用分配存在两大挑战：大词汇量下的长尾概率会污染熵值和梯度，且不确定性在正向与非正向优势轨迹中语义不同。我们提出非对称信用策略优化（ACPO），以顶词概率的补数作为模式局部代理替代全局熵。基于梯度分析，ACPO引入不匹配路由与饱和修正，使策略更新呈现期望的非对称形式：在成功轨迹上强调不确定决策，在失败轨迹上惩罚自信区域。理论上，ACPO局部保持优势方向并约束代理误差。在数学与编程推理基准测试（包括AIME 2025和HumanEval Pro）上的实验表明，ACPO始终优于熵感知方法（如80/20、GTPO）和强基线（如DAPO、SAPO）。
原文摘要 · Abstract (English)
Outcome-supervised reinforcement learning scales to verifiable reasoning tasks, but trajectory-level rewards assign the same outcome signal to all sampled tokens, overlooking their unequal contributions to the reasoning process. Entropy provides a natural indicator of the model's decision state, yet using it for token-level credit assignment presents two key challenges: long-tail probabilities in large vocabularies corrupt both entropy values and gradients, and uncertainty carries distinct semantics across positive- and non-positive-advantage trajectories. We propose Asymmetric Credit Policy Optimization (ACPO), which replaces global entropy with the complement of the top-token probability as a mode-local proxy. Guided by gradient analysis, ACPO incorporates mismatch routing and saturation correction to shape policy updates into the desired asymmetric form, emphasizing uncertain decisions on positive trajectories while penalizing confident regions on failed ones. Theoretically, ACPO locally preserves the advantage direction while bounding surrogate error. Experiments on mathematical and coding reasoning benchmarks, including AIME 2025 and HumanEval Pro, show that ACPO consistently outperforms both entropy-aware methods (e.g., 80/20, GTPO) and strong outcome-supervised RL baselines (e.g., DAPO, SAPO).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。