arXiv:2609.04773cs.LGcs.AI2026-09

提出持续教师锚定机制,提升工具使用智能体的生成稳定性与效率。

Persistent Teacher Anchoring for Tool-Using Agents

  • 学生生成时由教师逐轮验证,确保每步调用前经教师批准。
  • 在相同训练预算下,检索与感知任务中宏平均最佳4项得分分别提升2.5和2.8点。
  • 引入持久前瞻机制,使推理吞吐量提升24%,有效利用空闲计算资源。

在大模型后训练中,基于策略的知识蒸馏(OPKD)常用于为下游强化学习准备学生模型。每个状态中,学生匹配教师提供的下一个词分布。当回放进入教师未访问的状态时,教师-学生分布差距会累积。在工具使用场景中,该差距尤为严重,因为学生生成的调用提前执行,并影响后续输入前缀。现有方法虽通过提议-验证机制缓解生成漂移,但未涵盖工具执行环节。本文提出持续教师锚定(PTA),实现学生驱动但教师承诺的回放构建:仅当教师验证整轮内容后,调用才被提交至环境。将已验证块视为原子生成单元,引入持久前瞻机制,在固定验证器下推进未来样本并跨学生更新携带未完成样本。在Search-R1式检索与DeepEyes式感知强化学习任务中,采用PTA后的下游强化学习相比OPKD,宏平均最佳@4得分分别提升2.5和2.8点,同时前瞻机制使吞吐量提升24%。

原文摘要 · Abstract (English)

Distillation is common in LLM post-training, where on-policy knowledge distillation (OPKD) uses student-generated trajectories to prepare the student for downstream RL. At each state, the student matches a next-token distribution supplied by the teacher. As the rollout enters states the teacher would not visit, the teacher-student distribution gap can accumulate. In tool use, this gap becomes consequential because student-written calls execute before supervision and their observations shape later prefixes. Proposer-verifier generation addresses this drift by letting the teacher decide which student-proposed text is retained during generation. Existing formulations govern text but leave tool execution outside their scope. We propose Persistent Teacher Anchoring (PTA), a student-induced but teacher-committed rollout construction. PTA retains chunk-level verification and adds turn-level commitment, allowing a call to reach the environment only after the teacher has verified the entire turn. Treating verified chunks as atomic generation units, we introduce persistent lookahead, which fills idle rollout capacity by advancing future samples and carrying unfinished ones across student updates under the fixed verifier. Across Search-R1-style retrieval and DeepEyes-style perception RL, applying PTA before downstream RL improves macro best@4 by 2.5 and 2.8 points over OPKD under the same downstream RL budget, while lookahead improves throughput by 24%.

强化学习知识蒸馏工具使用生成控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。