让AI模型通过专家反馈自我进化，弱模型也能超越强教师。
Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
- 用带特权信息的AI教师持续指导学生模型改进决策能力。
- 弱模型（如Llama3-8B）性能超越GPT4-o等强教师模型。
- 适合想提升推理能力的模型开发者与研究者使用。
尽管大型语言模型在决策方面表现优异，但现有方法缺乏任务执行中从错误自动自我改进的机制。本文提出LEAP，一种迭代微调框架，利用AI专家教师的反馈持续提升LLM代理性能。核心思想是赋予专家教师特权状态——训练时可见但测试时隐藏的信息。这使得即使是弱专家也能提供精准指导，显著提升学生模型表现，且学生无需在测试时访问特权信息。我们在多样化的决策基准上评估LEAP，包括文本游戏（ALFWorld）、网页导航（WebShop）和交互式编码（Intercode Bash）。实验表明，LEAP优于行为克隆和ReAct基线；使弱学生模型（如Llama3-8B）超越强教师模型（GPT4-o）；并允许弱模型通过自身特权版本实现自我提升。我们还提供理论分析，证明LEAP的成功依赖于特权信息与学生可实现性之间的平衡，并在实验中加以验证。代码已开源。
原文摘要 · Abstract (English)
While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state -- information that is available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time. We evaluate LEAP on diverse decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves. We also provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with the student's realizability, which we empirically validate. Our code is available at https://leap-llm.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。