让大模型学会从失败中提取经验,无需实时查资料也能高效完成任务。
Fine-tuning with RAG for Improving LLM Learning of New Skills

- 通过分析任务失败,提取关键提示信息并生成改进的示范路径。
- 训练后模型在ALFWorld成功率达91%(基线79%),WebShop得分提升至72(基线61)。
- 适用于多步任务场景,适合希望减少依赖外部检索的部署者。
部署用于多步骤任务的大语言模型代理常因预条件未满足、重复指令或环境约束处理不当而失败。尽管检索增强生成(RAG)可在运行时提供指导以提升性能,但需维护外部知识库且每次部署都增加计算开销。我们提出一种简单流程,将推理时的检索转化为可内化的学习能力:(1) 从代理失败中提取紧凑可复用的提示;(2) 在每轮开始时通过一次检索生成改进的教师轨迹;(3) 用移除提示的轨迹训练学生模型,迫使其内化而非记忆。在两个交互式基准测试中——ALFWorld(家庭任务)和WebShop(在线购物)——蒸馏后的学生模型持续优于基线,ALFWorld成功率最高达91%(基线79%),WebShop得分提升至72(基线61),同时相比检索增强教师模型节省10%-60%的令牌数。该方法在不同模型规模(7B/14B参数)和代理架构(ReAct/StateAct)间具有良好泛化能力,证明可通过针对性微调有效内化检索优势,而无需永久性的运行时依赖。
原文摘要 · Abstract (English)
Large language model (LLM) agents deployed for multi-step tasks frequently fail in predictable ways: attempting actions with unmet preconditions, issuing redundant commands, or mishandling environment constraints. While retrieval-augmented generation (RAG) can improve performance by providing runtime guidance, it requires maintaining external knowledge databases and adds computational overhead at every deployment. We propose a simple pipeline that converts inference-time retrieval into learned competence through distillation. Our approach: (1) extracts compact, reusable hints from agent failures, (2) uses these hints to generate improved teacher trajectories via one-shot retrieval at episode start, and (3) trains student models on these trajectories with hint strings removed, forcing internalization rather than memorization. Across two interactive benchmarks, ALFWorld (household tasks) and WebShop (online shopping), distilled students consistently outperform baseline agents, achieving up to 91% success on ALFWorld (vs. 79% for baselines) and improving WebShop scores to 72 (vs. 61 for baselines), while using 10-60% fewer tokens than retrieval-augmented teachers depending on the environment. The approach generalizes across model scales (7B/14B parameters) and agent architectures (ReAct/StateAct), demonstrating that retrieval benefits can be effectively internalized through targeted fine-tuning without permanent runtime dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。