让大模型智能体在复杂任务中每步都得到精准奖励,提升成功率和执行准确率。
SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
- 通过分解最终奖励为每一步的进度贡献,实现细粒度反馈
- 在多个基准上平均提升成功率2.5%、定位准确率1.9%
- 适合需要多步决策与环境交互的智能体训练场景
强化学习(RL)在训练大语言模型(LLM)智能体完成复杂目标导向任务方面潜力巨大,但面临延迟奖励问题:反馈仅在任务完成后才出现,导致早期动作难以获得有效指导。本文提出一种通用的奖励重分配框架——逐步进度归因(SPA),其核心思想是任务成功源于各步骤的累积进展。通过训练一个进度估计器,将最终奖励分解为各步骤的增量贡献,并结合环境动作的锚定信号,生成细粒度中间奖励用于策略优化。在Webshop、ALFWorld和VirtualHome等常见智能体基准上的实验表明,SPA在成功率上平均提升2.5%,在定位准确率上平均提升1.9%,显著改善了强化学习训练中的中间奖励有效性。代码已开源。
原文摘要 · Abstract (English)
Reinforcement learning (RL) holds significant promise for training LLM agents to handle complex, goal-oriented tasks that require multi-step interactions with external environments. However, a critical challenge when applying RL to these agentic tasks arises from delayed rewards: feedback signals are typically available only after the entire task is completed. This makes it non-trivial to assign delayed rewards to earlier actions, providing insufficient guidance regarding environmental constraints and hindering agent training. In this work, we draw on the insight that the ultimate completion of a task emerges from the cumulative progress an agent makes across individual steps. We propose Stepwise Progress Attribution (SPA), a general reward redistribution framework that decomposes the final reward into stepwise contributions, each reflecting its incremental progress toward overall task completion. To achieve this, we train a progress estimator that accumulates stepwise contributions over a trajectory to match the task completion. During policy optimization, we combine the estimated per-step contribution with a grounding signal for actions executed in the environment as the fine-grained, intermediate reward for effective agent training. Extensive experiments on common agent benchmarks (including Webshop, ALFWorld, and VirtualHome) demonstrate that SPA consistently outperforms the state-of-the-art method in both success rate (+2.5\% on average) and grounding accuracy (+1.9\% on average). Further analyses demonstrate that our method remarkably provides more effective intermediate rewards for RL training. Our code is available at https://github.com/WangHanLinHenry/SPA-RL-Agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。