通过里程碑机制提升长序列语言智能体的训练效率与成功率。
Milestone-Guided Policy Learning for Long-Horizon Language Agents
- 以任务里程碑划分轨迹，分段进行奖励塑造，精准分配奖励信号。
- 在ALFWorld上成功率达92.9%，样本利用率从23.7%提升至82.0%。
- 适合需要长期决策的语言智能体训练，尤其适用于复杂任务场景。
长时序智能体任务要求语言智能体执行数十步连续决策，但使用强化学习训练仍面临挑战。我们识别出两个根本原因：信用误分配（早期正确动作因最终失败被惩罚）和样本低效（成功轨迹稀少导致学习信号几乎消失）。为此提出里程碑引导策略学习框架BEACON，利用长时序任务的组合结构实现精确信用分配。BEACON在里程碑边界分割轨迹，对各段内进行时间奖励塑造以认可局部进展，并在双尺度估计优势，防止远端失败干扰局部动作评估。在ALFWorld、WebShop和ScienceWorld上，BEACON始终优于GRPO和GiGPO。特别地，在长时序ALFWorld任务中，成功率高达92.9%，近乎翻倍于GRPO的53.5%，同时有效样本利用率从23.7%提升至82.0%。这些结果证明了以里程碑为锚点的信用分配是训练长时序语言智能体的有效范式。代码已开源：https://github.com/ZJU-REAL/BEACON。
原文摘要 · Abstract (English)
While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。