arXiv:2607.02502cs.LGcs.AI2026-07被引 3

通过动态调节教师指导,解决大模型推理训练中的信息泄露与探索抑制问题。

DemoPSD: Disagreement-Modulated Policy Self-Distillation

论文配图:DemoPSD: Disagreement-Modulated Policy Self-Distillation
图 1 · 摘自论文原文
  • 基于教师与学生分布差异,自适应融合两者输出,避免盲目模仿。
  • 在四个科学领域评测中超越GRPO和SDPO,且训练熵更高、泛化更强。
  • 适合需要稳定推理与跨域泛化的大型语言模型训练场景。

在线策略自蒸馏(OPSD)已成为训练大语言模型进行推理的实用方法,其中单个模型同时充当教师与学生,具有不同层次的信息访问能力。然而,近期研究发现,依赖特权信息的教师密集标记级监督会导致对域内模式的过拟合、抑制探索,并损害跨域泛化能力,更根本的问题是存在‘特权信息泄露’:学生习得仅在训练时可用的答案相关捷径。本文提出新框架DemoPSD,通过选择性采纳教师指导来解决上述问题。不拟合完整教师分布,而是引导学生向一个反KL质心目标逼近,该目标是教师与学生分布的加权几何组合,自然平衡学习教师与保留自身推理能力。通过测量两分布间差异,自适应控制每个词元位置的融合比例。理论证明,DemoPSD实现(1)泄露抑制,即有效缓解特权信息泄露;(2)探索保留,即在密集标记级蒸馏下维持探索能力。在四个科学领域的SciKnowEval上大量实验表明,DemoPSD优于GRPO和SDPO,保持更高训练熵,并稳健泛化至分布外的GPQA基准。

原文摘要 · Abstract (English)

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce **DemoPSD**, a novel framework that resolves such problems through the idea of *selective adoption of teacher guidance*. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a *reverse-KL barycenter target*, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves **(1)** *leakage attenuation*, i.e., effective mitigation of privileged information leakage; and **(2)** *exploration preservation*, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

大模型推理自蒸馏信息泄露泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。