为扩散语言模型设计新自蒸馏方法，提升训练效率。
Learning from the Self-future: On-policy Self-distillation for dLLMs
- 用自生成答案作后缀条件，让模型学习‘未来经验’
- 改词级监督为步骤级，适配扩散模型迭代去噪过程
- 仅需强化学习10%的训练步数，适合高效微调场景
在线策略自蒸馏（OPSD）在大语言模型后训练中表现优异，但其在扩散语言模型（dLLMs）中的应用尚未探索。现有OPSJ方法本质上依赖自回归结构，通过从左到右前缀条件注入特权信息，并以词级差异进行监督，这与dLLMs任意顺序生成的本质相冲突。本文提出d-OPSD，首个专为dLLMs设计的OPSD框架。核心贡献有二：其一，重构自教师构建方式，采用自生成答案作为后缀条件，使学生模型学习‘自我未来经验’；其二，将监督机制由词级转为步骤级，与dLLMs的迭代去噪过程对齐。在四个推理基准上的实验表明，d-OPSD持续优于强化学习与监督微调基线，具备更优样本效率，仅需约RLVR所需优化步数的10%，为dLLM后训练开辟了新路径。代码已开源：https://github.com/xingzhejun/d-OPSD。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing OPSD methods are inherently autoregressive-centric. They inject privileged information via left-to-right prefix conditioning with token-level divergence supervision, a design that fundamentally conflicts with the arbitraryorder generation of dLLMs. We introduce d-OPSD, the first OPSD framework tailored for dLLMs. Our approach makes two core contributions. First, we reframe self-teacher construction by using self-generated answers as suffix conditioning, enabling the student model to learn from "self future-experience" rather than privileged prefixes. Second, we shift supervision from token-level to step-level, aligning training with the iterative denoising process of dLLMs. Experiments across four reasoning benchmarks show that d-OPSD consistently outperforms RLVR and SFT baselines with superior sample efficiency, requiring only around 10% of the optimization steps by RLVR and opening a promising pathway for dLLM posttraining. The code is available at https://github.com/xingzhejun/d-OPSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。