通过调环境让AI代理学会复杂任务,少依赖专家数据。
Don't Just Fine-tune the Agent, Tune the Environment
- 让环境动态提供反馈,引导代理自主探索学习。
- 用400个任务实例即达基线水平,且泛化能力更强。
- 适合想提升智能体鲁棒性与数据效率的研究者。
大型语言模型(LLM)代理在复杂多轮工具使用任务中展现巨大潜力,但其发展常受限于高质量训练数据的极度稀缺。在合成数据上进行监督微调(SFT)会导致过拟合,而标准强化学习(RL)则面临严重的冷启动问题和训练不稳定性。为此,我们提出环境调优(Environment Tuning),一种新型训练范式,使代理能直接从问题实例中学习复杂行为,无需依赖预先收集的专家轨迹。该方法通过结构化课程、可操作的环境增强(提供纠正反馈)以及细粒度进度奖励,实现稳定高效的探索。仅使用伯克利函数调用排行榜(BFCL)中的400个问题实例,本方法不仅在分布内性能上达到强基线水平,更展现出卓越的分布外泛化能力,克服了基于SFT方法常见的性能坍塌问题。这项工作标志着从静态轨迹监督微调向动态环境驱动探索的范式转变,为训练更鲁棒、数据高效的智能体铺平道路。代码已公开于https://github.com/inclusionAI/AWorld-RL/tree/main/EnvTuning。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents show great promise for complex, multi-turn tool-use tasks, but their development is often hampered by the extreme scarcity of high-quality training data. Supervised fine-tuning (SFT) on synthetic data leads to overfitting, whereas standard reinforcement learning (RL) struggles with a critical cold-start problem and training instability. To address these challenges, we introduce $\textbf{Environment Tuning}$, a novel training paradigm that enables agents to learn complex behaviors directly from problem instances without relying on pre-collected expert trajectories. $\textbf{Environment Tuning}$ orchestrates this learning process through a structured curriculum, actionable environment augmentation that provides corrective feedback, and fine-grained progress rewards to ensure stable and efficient exploration. Using only 400 problem instances from Berkeley Function-Calling Leaderboard (BFCL) benchmark, our method not only achieves competitive in-distribution performance against strong baselines but also demonstrates superior out-of-distribution generalization, overcoming the performance collapse common to SFT-based approaches. Our work presents a paradigm shift from supervised fine-tuning on static trajectories to dynamic, environment-based exploration, paving the way for training more robust and data-efficient agents. The code is available at https://github.com/inclusionAI/AWorld-RL/tree/main/EnvTuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。