解决大模型长序列任务中奖励稀疏问题，提升决策信用分配的可靠性。
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
- 通过动作归类与方差门控校准每步奖励，避免偶然高分误导训练
- 在ALFWorld和WebShop上相比GiGPO提升5.2~7.3成功率点
- 无需价值函数，仅增加0.1%计算开销，适合资源受限的长周期智能体训练
长时序大模型智能体需要强化学习方法在稀疏延迟奖励下对中间决策进行信用分配。现有基于组的方法如GiGPO通过重复锚点状态构造步骤级优势，但我们在有限采样下发现密集信用可能统计不可靠：罕见但幸运的动作可能获得过高的优势，导致锚点偏差和后期训练振荡。为此提出证据校准策略优化（ECPO），一种无需价值函数的策略优化算法，在策略更新前校准步骤级信用。ECPO结合证据校准动作优势（按标准动作分组并压缩低频估计）与方差门控信用加权（抑制由组内噪声主导的锚点状态）。在ALFWorld与WebShop上使用Qwen2.5-1.5B/7B的实验表明，ECPO持续优于强基线，相较GiGPO在两个数据集上分别提升+5.2/+7.3成功率点，额外优势计算开销仅0.1%。
原文摘要 · Abstract (English)
Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards. Recent group-based methods such as GiGPO improve over GRPO by constructing step-level advantages at repeated anchor states. However, we show that such dense credit can be statistically unreliable: under limited rollouts, rare but lucky actions may receive overly large advantages, producing divergent anchor bias and late-stage training oscillation. We propose Evidence-Calibrated Policy Optimization (ECPO), a critic-free policy optimization algorithm that calibrates step-level credit before policy updates. ECPO combines Evidence-Calibrated Action Advantage, which groups rollouts by canonical actions and shrinks low-count estimates, with Variance-Gated Credit Weighting, which suppresses anchor states dominated by within-action noise. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B show that ECPO consistently outperforms strong baselines, improving GiGPO by +5.2/+7.3 success points on ALFWorld/WebShop with Qwen2.5-1.5B while adding only 0.1% additional advantage-computation overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。