arXiv:2608.14945cs.AIcs.CL2026-08

用影响校准提升语言智能体自蒸馏的监督效果,避免无效指导。

Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL

论文配图:Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
图 1 · 摘自论文原文
  • 基于教师输出扰动测量令牌重要性,动态调整监督权重。
  • 在7B模型上实现96.1%阿尔福世界成功率,比纯信任分配提升显著。
  • 适合追求高精度指令执行的自主智能体研究者使用。

在策略自蒸馏(OPSD)中,语言智能体通过自身轨迹获得来自特权教师的密集令牌级监督。现有方法主要依据教师信任度分配监督,但信任度无法反映某令牌强调是否有助于当前策略目标,存在信任-效用错配问题。本文提出影响校准自蒸馏(ICSD),对每个被监督令牌,衡量其重要性加权的强化学习代理贡献对教师引导输出扰动的一阶响应。批量自适应校准将这一非平稳信号转化为有界分配权重,同时保持每轮动作中辅助损失总量不变。该权重仅作用于蒸馏损失,无需额外模型前向传播。在ALFWorld、WebShop和Search-QA任务上,ICSD在GRPO与GiGPO框架下均优于纯信任分配,覆盖1.5B至7B两个模型家族。7B模型达到96.1%的ALFWorld成功率达与93.1的WebShop得分。冻结批次分析显示,ICSD将目标对立令牌的教师支持质量从60.1%降至37.8%,并使余弦兼容性提升0.192。相关代码库已公开。

原文摘要 · Abstract (English)

On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.

自蒸馏强化学习智能体监督优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。