用大模型反馈训练智能体，无需专家数据也能提升性能。
Training Agents with Weakly Supervised Feedback from Large Language Models
- 用批判性大模型筛选优质行为轨迹，弱监督优化智能体
- 在API-bank上表现接近GPT-4，参数量远少于GPT-4
- 适合缺乏专家数据的复杂任务场景，如自动化工具调用
大型语言模型（LLM）为通过迭代环境交互完成复杂任务的智能体提供了潜在基础。现有方法要么要求智能体模仿专家提供的轨迹，要么依赖明确的环境反馈进行强化学习，限制了其在游戏或代码生成等特定场景的应用。本文提出一种基于批判性大模型弱监督信号的新训练方法，无需专家轨迹或确定性反馈。智能体通过环境交互生成轨迹，随后由一个批判性大模型选出优质轨迹，用于更新智能体，使其在下一轮生成更优轨迹。在API-bank数据集上的大量测试表明，该方法能持续提升智能体能力，性能与GPT-4相当，尽管使用的是参数量远小于GPT-4的开源模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) offer a promising basis for creating agents that can tackle complex tasks through iterative environmental interaction. Existing methods either require these agents to mimic expert-provided trajectories or rely on definitive environmental feedback for reinforcement learning which limits their application to specific scenarios like gaming or code generation. This paper introduces a novel training method for LLM-based agents using weakly supervised signals from a critic LLM, bypassing the need for expert trajectories or definitive feedback. Our agents are trained in iterative manner, where they initially generate trajectories through environmental interaction. Subsequently, a critic LLM selects a subset of good trajectories, which are then used to update the agents, enabling them to generate improved trajectories in the next iteration. Extensive tests on the API-bank dataset show consistent improvement in our agents' capabilities and comparable performance to GPT-4, despite using open-source models with much fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。