自适应切换执行器，让智能体训练更稳更快
DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation
- 根据动作概率差异动态切换学生与教师执行器
- 在14项任务中表现优于5个基线，9次轨迹更短
- 适合多轮对话场景的强化学习训练，提升稳定性
在线策略蒸馏（OPD）通过让学生模型基于自身采样进行训练来减少暴露偏差。然而，在多轮智能体场景中，早期的学生错误可能导致轨迹偏离教师熟悉的领域。现有课程学习方法虽能根据训练进度调节教师支持程度，却无法判断何时需要干预。为此，我们提出 DASH-OPD：一种具备差异常识与滞后的在线策略蒸馏方法，是首个可双向自适应切换执行器的智能体化 OPD 方法。每轮中，该方法计算两个执行器在动作词上的平均对数概率比作为差异度量；学生轮次的师生概率比构成漂移信号，教师轮次的反向比值构成恢复信号。这些信号经归一化后在多轮中累积为漂移与恢复证据，当证据超过阈值时触发执行器切换。多轮累积使切换具有滞后性，有效防止因瞬时波动导致的频繁切换。在 WebShop、ALFWorld 与 ScienceWorld 三个数据集上，针对两种学生模型规模，DASH-OPD 在全部 14 项任务性能比较中胜过五个基线，并在 9/10 的回合数对比中实现最短轨迹，展现出最优的整体性能-效率权衡。本文为进行中工作，代码、训练日志与模型检查点将后续发布。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains student models on their own rollouts to reduce exposure bias. However, in multi-turn agent scenarios, early student errors can lead a trajectory away from the teacher's familiar domain. Existing curriculum learning methods regulate how much teacher support is used according to training progress, but cannot determine when it is needed. In light of this, we propose DASH-OPD, Discrepancy-Aware Switching with Hysteresis for OPD, the first agentic OPD method that can switch executors adaptively and bidirectionally. On each turn, DASH-OPD calculates a mean log-probability ratio between the two executors over action tokens as their discrepancy. Student-to-teacher ratios on student turns form drift signals, while teacher-to-student ratios on teacher turns form recovery signals. These signals are normalized and accumulated over multiple turns into drift and recovery evidence. DASH-OPD switches executors when the evidence exceeds its corresponding switching threshold. This multi-turn accumulation makes the switching hysteretic, preventing high-frequency switches caused by transient fluctuations. Across WebShop, ALFWorld, and ScienceWorld at two student-model scales, DASH-OPD outperforms five baselines in all 14 task-performance comparisons while yielding the shortest trajectories in nine of ten turn-count comparisons, offering the strongest overall performance-efficiency trade-off. This paper is a work in progress. Code, training logs, and model checkpoints will be released later.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。