arXiv:2606.30626cs.AI2026-06被引 4

提出双策略强化学习蒸馏方法,解决特权信息带来的虚假迁移问题。

DOPD: Dual On-policy Distillation

论文配图:DOPD: Dual On-policy Distillation
图 1 · 摘自论文原文
  • 根据优势差距动态分配令牌级监督信号,由教师或学生自身提供。
  • 在大语言模型和视觉语言模型上均显著超越传统方法。
  • 特别适合需要稳定性和持续学习的复杂任务场景。

在线策略蒸馏(OPD)通过密集的令牌级信号监督学生采样轨迹,实现更强的能力迁移。为提升蒸馏性能,一个直观方向是向教师或学生引入特权信息。然而,这种额外输入会引发我们称之为‘特权幻觉’的问题:混淆了学生应弥补的能力差距与仅可模仿无法复制的信息不对称差距。该问题在令牌级监督非均匀性下进一步加剧,仅有少量关键令牌携带核心能力信号。为此,我们提出DOPD——一种基于优势感知的双蒸馏范式,根据教师与学生间的优势差距及相对概率,动态路由令牌级监督信号。每个令牌从教师或学生自身接收不同强度、目标和策略的监督,既传递可信能力,又获得辅助信号,有效缓解特权幻觉。在大语言模型(LLM)与视觉语言模型(VLM)上的大量实验表明,DOPD始终优于基线方法。稳定性、鲁棒性、持续学习及分布外任务结果进一步验证其优越性。

原文摘要 · Abstract (English)

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

强化学习模型蒸馏大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。