arXiv:2607.04428cs.CLcs.AI2026-07

让扩散语言模型自我蒸馏,提升推理与生成能力。

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

论文配图:dOPSD: On-Policy Self-Distillation for Diffusion Language Models
图 1 · 摘自论文原文
  • 用学生模型自身解码轨迹生成教师信号,实现在线自蒸馏。
  • 在Dream和LLaDA数据集上显著提升数学推理与代码生成性能。
  • 无需外部标签,适合需强推理的扩散语言模型优化场景。

扩散大语言模型(dLLMs)通过迭代去噪生成文本,是自回归模型的并行替代方案,但其后训练阶段强化推理能力仍具挑战:监督微调为离策略方法,存在暴露偏差;强化学习仅提供稀疏的序列级奖励,且依赖可计算的序列似然。在策略自蒸馏(OPSD)中,模型同时作为学生与教师,提供密集的、逐标记的、在策略的监督信号,但其效果依赖于教师获得特权信息(PI),通常为推理时不可用的实例特定真实参考,导致学生蒸馏出弱化的无特权共识策略,改进有限。本文提出dOPSD,从学生自身的去噪轨迹中直接推导教师的特权信息,通过后续更解码的步骤评估掩码位置,而非依赖外部标签,使教师优势源自模型自身解码过程;在Dream和LLaDA数据集上,dOPSD显著提升领域内数学推理与领域外代码生成能力,优于监督及在策略基线方法。

原文摘要 · Abstract (English)

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.

扩散模型自蒸馏语言模型推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。