提出可确定性采样的掩码扩散新方法,推理速度提升16倍。
Latent Shadows: The Gaussian-Discrete Duality in Masked Diffusion
- 基于高斯-离散对偶性,建立掩码扩散的确定性轨迹构造机制。
- 实现16倍推理加速,生成质量不下降,超越传统随机采样。
- 适合追求高效文本生成的研究者与工业应用开发者。
掩码离散扩散是高质量语言建模的主流范式,通过迭代掩码化词元实现,但其推理效率受限于缺乏确定性采样工具。尽管扩散对偶性支持均匀模型的确定性蒸馏,但这些方法通常在掩码模型上表现较差,且依赖复杂积分算子。相反,现有掩码方法普遍假设不存在确定性路径,只能依赖随机蒸馏。为此,本文建立了显式的掩码扩散对偶性,证明掩码过程是连续高斯过程通过新颖的最大值索引保持机制投影所得。进一步提出掩码一致性蒸馏(MCD),基于该对偶性解析构建一致性蒸馏所需的确定性耦合轨迹,无需数值微分方程求解器。该方法严格优于以往随机蒸馏,实现16倍推理速度提升,且生成质量无损。研究不仅为掩码与连续扩散提供了理论连接,更释放了一致性蒸馏在高性能离散生成中的潜力。代码已公开于https://anonymous.4open.science/r/MCD-70FD。
原文摘要 · Abstract (English)
Masked discrete diffusion is a dominant paradigm for high-quality language modeling where tokens are iteratively corrupted to a mask state, yet its inference efficiency is bottlenecked by the lack of deterministic sampling tools. While diffusion duality enables deterministic distillation for uniform models, these approaches generally underperform masked models and rely on complex integral operators. Conversely, in the masked domain, prior methods typically assume the absence of deterministic trajectories, forcing a reliance on stochastic distillation. To bridge this gap, we establish explicit Masked Diffusion Duality, proving that the masked process arises as the projection of a continuous Gaussian process via a novel maximum-value index preservation mechanism. Furthermore, we introduce Masked Consistency Distillation (MCD), a principled framework that leverages this duality to analytically construct the deterministic coupled trajectories required for consistency distillation, bypassing numerical ODE solvers. This result strictly improves upon prior stochastic distillation methods, achieving a 16$\times$ inference speedup without compromising generation quality. Our findings not only provide a solid theoretical foundation connecting masked and continuous diffusion, but also unlock the full potential of consistency distillation for high-performance discrete generation. Our code is available at https://anonymous.4open.science/r/MCD-70FD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。