arXiv:2607.04574cs.LGcs.AI2026-07被引 1

用少量教师步骤在学习者上下文中生成数据,比长文本或严格筛选更省钱高效。

A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training

论文配图:A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training
图 1 · 摘自论文原文
  • 在学习者遇到的上下文中,仅让教师输出几步响应
  • 相同预算下,成功率比纯行为克隆提升,且优于长续写或过滤方案
  • 适合追求低成本高效训练的LLM代理开发者

对于大模型智能体,监督微调不仅取决于教师标签的质量,还取决于标签所依赖的交互上下文。纯行为克隆使用完整的教师示范,导致训练时教师诱导的上下文与测试时学生诱导的上下文不匹配。近期工作通过在学生达到的上下文中查询教师来缓解这一问题,常配合复杂过滤机制。本文将在线策略数据构建视为预算分配问题:在监督资源固定的情况下,应将教师输出用于更多完整示范、更长续写、结果过滤,还是更广泛覆盖学习者引发的上下文?我们通过展开策略、切换时间分布、续写时长、过滤规则及两项互补成本(过滤前的教师推理开销和用于SFT的监督保留量)形式化该设计空间。在HotpotQA、ALFWorld和Terminal-Bench-Dev上,对学习者上下文采用有限未过滤的教师续写,在相同预算下优于纯行为克隆。在HotpotQA和ALFWorld上,少量步骤的续写在性能上可媲美甚至超过成功过滤和关键上下文过滤方案。结果表明,将少量教师步骤放置于学习者触发的上下文中,是一种比长续写或深度筛选更高效的监督分配方式。

原文摘要 · Abstract (English)

For LLM agents, supervised fine-tuning is not only about teacher labels' quality, but also about which interaction contexts those labels condition on. Pure behavioral cloning uses full teacher demonstrations, creating a mismatch between teacher-induced contexts seen in training and student-induced contexts encountered at test time. Recent work addresses this mismatch by querying a teacher at contexts reached by the student, often with increasingly elaborate filtering of the teacher's continuations. We instead frame on-policy data construction as a budget-allocation problem: under matched supervision resources, should teacher output be spent on more start-to-finish demos, longer continuations, outcome filtering, or broader coverage of learner-induced contexts? We formalize this design space through the rollout policy, switch-time distribution, continuation horizon, filtering rules, and two complementary costs: teacher inference generated before filtering and teacher supervision retained for SFT. Across HotpotQA, ALFWorld, and Terminal-Bench-Dev, bounded unfiltered teacher continuations at learner-induced contexts improve over pure behavioral cloning at matched budgets. On HotpotQA and ALFWorld, where we run the full comparison, few-step continuations match or exceed success-filtered and critical-context-filtered alternatives. Our findings suggest that a few teacher steps, placed at learner-induced contexts, can be a more cost-efficient supervision allocation than longer or more heavily curated teacher completions.

LLM代理数据增强成本优化监督微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。