通过事后视角降低长序列任务训练方差，提升语言智能体性能
From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
- 将当前策略与事后视角分布映射到意图空间，利用沃尔德斯坦距离提取低方差信号
- 在多个长时序任务上实现稳定性能提升，方差显著降低
- 适合需要复杂推理和长期规划的语言智能体研究者使用
强化学习（RL）已成为提升大语言模型（LLMs）在复杂任务上表现的主流方法。然而，现有方法在训练长时序交互智能体时仍面临挑战，主要瓶颈在于难以区分不同动作在长期交互中的贡献，导致优化方差过高。为此，我们提出一种新的策略梯度方法——事后策略优化（Hindsight Policy Optimization, HPO），将当前策略分布与事后分布投影至意图空间，并从中提取沃尔德斯坦距离作为低方差学习信号。理论与实证均表明，在意图空间中聚合语义相似的状态与动作，可获得有界方差估计器，从而实现更稳定的策略性能提升。代码已公开。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a widely adopted technique for improving large language models (LLMs) on complex tasks. Despite this progress, existing RL methods still face challenges in training agents with longer-horizon interactions. One major bottleneck is distinguishing the contribution of different actions in long-horizon interaction, leading to high optimization variance. To address this, we introduce a novel policy gradient method, Hindsight Policy Optimization (HPO), that projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them. We theoretically and empirically show that aggregating semantically similar states and actions in the intent space yields a bounded-variance estimator and improves policy performance stably. Our code is available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。