用事后反思提升大模型长任务决策能力，效果显著优于现有方法。
Hindsight Credit Assignment for Long-Horizon LLM Agents
- 让大模型自己当裁判，事后回溯优化每一步的奖励判断。
- 在WebShop和ALFWorld上分别比基线提升7.7%和13.8%成功率。
- 适合需要长期规划与精准决策的智能体应用，如复杂任务自动化。
大型语言模型（LLM）代理在长时序多步任务中常因奖励稀疏而面临严重的信用分配挑战。现有无价值方法（如GRPO）存在两个根本瓶颈：步骤级Q值估计不准确、中间状态价值基线错位。为此，我们提出HCAPO，首个将事后信用分配引入LLM代理的框架。HCAPO利用大模型自身作为事后评判者，通过事后推理精炼步骤级Q值；同时，其多尺度优势机制有效弥补关键决策状态下的不准确价值基线。在WebShop和ALFWorld等三个挑战性基准上的评估表明，HCAPO持续优于当前最优强化学习方法。尤其在使用Qwen2.5-7B-Instruct模型时，于WebShop上成功率达7.7%提升，于ALFWorld上达13.8%提升。结果表明，HCAPO显著提升了探索效率，促进决策简洁化，并保障了复杂长时序任务中的可扩展性。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents often face significant credit assignment challenges in long-horizon, multi-step tasks due to sparse rewards. Existing value-free methods, such as Group Relative Policy Optimization (GRPO), encounter two fundamental bottlenecks: inaccurate step-level Q-value estimation and misaligned value baselines for intermediate states. To address these limitations, we introduce HCAPO, the first framework to integrate hindsight credit assignment into LLM agents. HCAPO leverages the LLM itself as a post-hoc critic to refine step-level Q-values through hindsight reasoning. Furthermore, HCAPO's multi-scale advantage mechanism effectively supplements the inaccurate value baselines at critical decision states. Evaluations across three challenging benchmarks, including WebShop and ALFWorld, demonstrate that HCAPO consistently outperforms state-of-the-art RL methods. Notably, HCAPO achieves a 7.7% improvement in success rate on WebShop and a 13.8% on ALFWorld over GRPO using the Qwen2.5-7B-Instruct model. These results indicate that HCAPO significantly enhances exploration efficiency, promotes concise decision-making, and ensures scalability in complex, long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。