提出新方法让扩散语言模型少步生成更准,关键在动态压缩推理路径。
OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models

- 用学生自身轨迹采样状态,结合教师判断未来候选
- 自适应选择最长保真路径,确保结果与教师一致
- 无需真实答案即可训练,适合高效代码生成任务
扩散语言模型(dLLMs)可并行预测多个词元,但高精度生成仍需多次去噪迭代。少步蒸馏通过将多步教师过程压缩为单步学生转移来加速解码。然而,现有方法基于非策略轨迹构建监督信号,在推理时,学生早期并行决策会改变后续上下文,导致实际访问状态偏离受训状态——尤其在压缩最激进时。策略内蒸馏可缓解此偏差,但如何确定每步推进距离仍存挑战:仅匹配下一步限制压缩,随意合并未来动作又可能破坏中间依赖。为此,本文提出OPTD,即基于一致性引导的自适应压缩策略内转移蒸馏。它从少步学生自身轨迹中采样部分状态,利用冻结的仅问答教师识别结果对齐的未来候选,并按当前状态置信度排序。随后选择最长前缀,其联合承诺保持教师回放结果一致。集合瓶颈目标推动每个验证过的未来候选达到解码释放阈值,冻结教师KL锚点正则化所有其他活跃位置。训练与目标构造均不依赖真实回答。在四个数学推理与代码生成基准上,OPTD持续提升质量-效率权衡,在评估的少步基线中取得最优整体质量约束下的AUP表现。
原文摘要 · Abstract (English)
Diffusion language models (dLLMs) can predict many tokens in parallel, but accurate generation still requires many iterative denoising steps. Few-step distillation accelerates decoding by compressing multiple teacher steps into a single student transition. However, existing methods construct supervision on off-policy trajectories. At inference, the student's early parallel commitments alter the context of later predictions, so the states it actually visits drift away from the supervised ones--precisely when step compression is most aggressive. On-policy distillation is a natural remedy for this mismatch, but it leaves open how far each transition should advance: matching only the teacher's next action limits compression, while indiscriminately merging future actions can violate intermediate dependencies. To address this limitation, we propose OPTD, On-Policy Transition Distillation with consistency-guided adaptive compression. It samples partial states from the few-step student's own trajectories, uses a frozen, question-only teacher to identify outcome-aligned future candidates, and orders them by current-state confidence. The method then selects the longest prefix whose joint commitment preserves the teacher's rollout outcome. A set-bottleneck objective promotes every verified future candidate to the decoder's release threshold, while a frozen-teacher KL anchor regularizes all other active positions. Neither target construction nor training uses a gold response. Across four mathematical reasoning and code-generation benchmarks, OPTD consistently improves the quality--efficiency trade-off and attains the strongest overall quality-constrained AUP among the evaluated few-step baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。