只训练专家轨迹中的关键步骤，让大模型代理更通用、更高效。
ATLaS: Agent Tuning via Learning Critical Steps
- 识别专家轨迹中关键步骤，仅对这些步骤微调模型。
- 仅用30%关键步骤训练，性能超越全轨迹微调和现有开源代理。
- 保留基础模型能力，适合需要泛化能力的复杂任务场景。
大型语言模型（LLM）代理在多领域任务中展现出卓越的泛化能力。现有代理微调方法通常对完整专家轨迹进行监督微调，但完整轨迹的行为克隆会引入专家偏差，并削弱模型在未覆盖状态下的泛化能力。此外，规划、复杂推理和战略决策等关键步骤对代理成功至关重要，学习这些步骤是提升LLM代理的核心。为实现更有效、高效的代理微调，我们提出ATLaS，通过识别专家轨迹中的关键步骤，仅对这些步骤进行微调，从而降低训练成本。通过将训练重点聚焦于少数关键步骤，该方法降低了过拟合整个轨迹的风险，促进了跨环境与任务的泛化。在大量实验中，仅使用ATLaS筛选出的30%关键步骤进行微调的LLM，性能优于全轨迹微调的LLM及近期开源的LLM代理。ATLaS还能保持并提升基础LLM作为通用代理在多样化环境中的交互能力。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents have demonstrated remarkable generalization capabilities across multi-domain tasks. Existing agent tuning approaches typically employ supervised finetuning on entire expert trajectories. However, behavior-cloning of full trajectories can introduce expert bias and weaken generalization to states not covered by the expert data. Additionally, critical steps, such as planning, complex reasoning for intermediate subtasks, and strategic decision-making, are essential to success in agent tasks, so learning these steps is the key to improving LLM agents. For more effective and efficient agent tuning, we propose ATLaS that identifies the critical steps in expert trajectories and finetunes LLMs solely on these steps with reduced costs. By steering the training's focus to a few critical steps, our method mitigates the risk of overfitting entire trajectories and promotes generalization across different environments and tasks. In extensive experiments, an LLM finetuned on only 30% critical steps selected by ATLaS outperforms the LLM finetuned on all steps and recent open-source LLM agents. ATLaS maintains and improves base LLM skills as generalist agents interacting with diverse environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。