用离散版的矩匹配，让离散扩散模型蒸馏后更快更优。
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
- 借鉴连续扩散模型的成功经验，提出离散矩匹配蒸馏方法。
- 蒸馏后生成质量高、多样性好，采样步数足够时性能超越教师模型。
- 适用于文本与图像任务，尤其适合追求高效生成的场景。
目前离散扩散模型的蒸馏仍面临挑战。相比之下，连续扩散领域已有多种成熟蒸馏方法，可将采样步骤大幅减少。本文提出的离散矩匹配蒸馏（D-MMD）方法，借鉴了连续域中已被证明有效的思想。与以往容易坍缩的离散蒸馏方法不同，D-MMD 在采样步数充足时能保持高质量和多样性。该方法在文本与图像数据集上均验证有效，且新蒸馏出的生成器甚至可超越其教师模型。
原文摘要 · Abstract (English)
It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。