用事后目标重标注提升大模型智能体的长程任务表现
Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations
- 让辅助大模型分析智能体轨迹，提取真实达成的目标并重标注
- 在ALFWorld上仅用1/4真实演示数据，效果超过全量数据基线
- 适合长周期、目标多样任务的高效训练，兼容现有微调流程
大型语言模型智能体在部分可观测、长周期环境中运行，获取监督信号仍是主要瓶颈。本文提出事后监督学习（HSL），利用现有后训练方法忽略的监督信号：智能体轨迹中隐含的意外但成功的自然语言目标。具体而言，引入一个辅助大模型对每个完成的轨迹进行审查，并将其重标注为智能体实际达成的所有自然语言目标。随后将轨迹与重标注目标配对，用于进一步微调。为缓解重标注数据中的次优问题，提出无关动作掩码和样本重加权两种学习技术。实验表明，HSL具有灵活性且可兼容现有后训练流程，在SFT和DPO上均取得提升，尤其在长周期、目标空间多样的任务上增益更大。此外，HSL具备高样本效率：在ALFWorld上，仅使用四分之一的真实演示数据即超越使用完整数据集的基线。
原文摘要 · Abstract (English)
Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。