arXiv:2605.12913cs.LG2026-05被引 1

用DAgger方法训练大模型智能体,兼顾真实状态分布与丰富反馈。

Revisiting DAgger in the Era of LLM-Agents

  • 通过师生策略逐轮交替生成轨迹,缓解策略偏差问题
  • 4B模型达27.3%准确率,超多数8B级系统
  • 适合长序列任务的智能体训练,尤其软件工程场景

长时序语言模型智能体依赖多轮交互学习,早期错误会改变后续状态分布,导致轨迹偏离。现有方法各有不足:监督微调虽提供密集教师指导,但因使用离线策略轨迹而存在协变量偏移;强化学习虽基于在线采样避免此问题,却仅有稀疏结果反馈。本文重新引入数据集聚合(DAgger)用于多轮语言模型智能体:算法通过学生与教师策略的逐轮插值生成轨迹,再以教师提供的监督标签训练学生模型。直接与环境交互使模型接触部署中真实可能的状态,有效缓解协变量偏移;同时,学生通过模仿教师行为获得丰富学习信号。实验验证该方法优势:在SWE-bench Verified上,4B和8B规模的学生模型分别比最强后训练基线提升+3.9和+3.6点。4B模型达到27.3%,超越多数公开8B级系统;8B模型达29.8%,超过SWE-Gym-32B,并接近32B级强模型。在保留测试集上也保持一致增益,表明DAgger对现代长时序语言模型智能体的有效性。

原文摘要 · Abstract (English)

Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory. Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision but suffers from covariate shift because it is trained on off-policy teacher trajectories; while reinforcement learning with verifiable rewards avoids this off-policy mismatch by learning from on-policy rollouts but with only sparse outcome feedback. We address this dilemma by revisiting Dataset Aggregation (DAgger) for multi-turn LM agents: the algorithm collects trajectories through a turn-level interpolation of student and teacher policies, and the student is then trained on these trajectories using supervised labels provided by the teacher. By directly interacting with environments, we expose the model to realistic states likely to be encountered during deployment, thereby effectively mitigating covariate shift. Besides, since the student is learned by mimicking the teacher's behavior, it receives rich feedback during learning. To demonstrate DAgger enjoys the benefits of both worlds, we tested the algorithm to train a software-engineering agent with 4B- and 8B-scale student models. On SWE-bench Verified, our DAgger-style training improves over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B. The resulting 4B agent reaches 27.3%, outperforming representative published 8B SWE-agent systems, while the 8B agent achieves 29.8%, surpassing SWE-Gym-32B and coming within 5 points of stronger 32B-scale agents. Together with consistent gains on the held-out SWE-Gym split, these results suggest the effectiveness of DAgger for modern long-horizon LM agents.

大模型智能体长序列生成DAgger软件工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。