解决大模型智能体长程任务中的奖励不一致问题
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
- 通过成功子轨迹反推失败步骤的合理奖励
- 在ALFWorld上达到93.7%成功率，关键提升在复杂多步任务
- 适合需要稳定学习路径的长程决策场景
基于组的强化学习通过轨迹最终结果分配每一步的信用，导致语义相近的中间步骤因最终成败获得相反奖励，造成梯度冲突并浪费失败轨迹中的部分正确进展。为此，我们提出语义一致性策略优化（SCPO），一种无需价值函数的奖励塑形方法，通过同一组轨迹中成功的子轨迹来恢复失败步骤的步级信用。具体地，将每个失败步骤与成功子轨迹对比，为沿该成功路径的新进展赋予正向步级奖励。在ALFWorld和WebShop任务上，SCPO表现优于或持平强基线，在15亿参数规模下，ALFWorld成功率达93.7±4.1%，WebShop达74.8±2.0%，尤其在最难的多步任务上提升显著。
原文摘要 · Abstract (English)
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。