让大模型失败轨迹变有用，用回溯重标注提升训练效率
AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling
- 将失败的对话轨迹重新标注为可学习的成功样本
- 在多个数据集上提升7.6%-11.4%性能，样本效率翻倍
- 适合想低成本提升智能体训练效果的研究者
LLM智能体训练常丢弃失败轨迹，尽管GPT-4o在WebArena上成功率仅14-20%，ToolBench pass@1低于55%，即使专业系统也仅达50-65%。我们提出AgentHER，将回溯经验重放（HER）适配至自然语言轨迹：失败轨迹往往可作为达成其他目标的有效示范。通过四阶段流程（失败分类、结果提取、基于信心门控的LLM引导重标注、数据封装），将废弃失败转化为SFT、DPO和ShareGPT数据。在任务不重叠的严格测试协议下，相比仅用成功轨迹的SFT，AgentHER在四个模型族（GPT-4o、Qwen2.5-72B/7B、LLaMA-3.1-8B）上提升7.6-11.4%，实现2倍样本效率，并超越最强经验中心基线（Agent Workflow Memory）3.0-6.2%。引入失败严重性加权与跨模型多裁判验证（gpt-4o-mini与Qwen2.5-72B-Instruct），将标签噪声从5.9%降至2.9%，人类评估重标注准确率达WebArena 97.1%、ToolBench 96.0%。全系统成本审计显示，处理3,000条轨迹耗时2.98与26分钟，每条有效样本成本仅1.4×10⁻³。
原文摘要 · Abstract (English)
LLM-agent training pipelines routinely discard failed trajectories even though GPT-4o achieves only 14-20% on WebArena and below 55% pass@1 on ToolBench; even specialised systems at 50-65% leave the majority of trajectories unused. We introduce AgentHER, which recovers this lost signal by adapting Hindsight Experience Replay (HER) to natural-language agent trajectories: a trajectory that fails goal A is often a correct demonstration for an achievable alternative goal B. AgentHER realises this through a four-stage pipeline (failure classification, outcome extraction, LLM-guided relabeling with confidence gating, and data packaging) that converts discarded failures into SFT, DPO, and ShareGPT training data. On WebArena and ToolBench under a strict task-disjoint held-out protocol, AgentHER improves over success-only SFT by +7.6-11.4% across four model families (GPT-4o, Qwen2.5-72B/7B, LLaMA-3.1-8B), achieves 2x sample efficiency, and beats the strongest experience-centric baseline (Agent Workflow Memory) by +3.0-6.2%. Two robustness mechanisms, failure-severity weighting and cross-model multi-judge verification (gpt-4o-mini paired with Qwen2.5-72B-Instruct), reduce label noise from 5.9% to 2.9% and raise human-rated relabeling precision to 97.1% on WebArena and 96.0% on ToolBench. A full system-cost audit shows the entire relabeling pipeline costs 2.98 and 26 wall-clock minutes for 3,000 trajectories, i.e. 1.4 x 10^-3 per accepted pair. Code: https://github.com/alphadl/AgentHER
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。