研究扩散模型蒸馏如何降低记忆过拟合，提升泛化能力。
On the Memorization of Consistency Distillation for Diffusion Models
- 通过一致性蒸馏抑制记忆相关不稳定特征方向
- 蒸馏后学生模型记忆数据程度显著下降，生成质量保持甚至提升
- 适合关注模型泛化与安全性的研究者和开发者
扩散模型是现代生成建模的核心，理解其在记忆与泛化间的平衡对可靠部署至关重要。近期研究表明，扩散模型的记忆行为受训练动态影响，泛化与记忆在不同训练阶段显现。然而，实际部署的扩散模型常进一步进行蒸馏，这一额外训练阶段对记忆的影响尚不明确。本文以一致性蒸馏为典型框架，分析其如何重塑扩散模型的记忆行为。实证发现，当教师模型已产生数据记忆时，一致性蒸馏能显著降低学生模型继承的记忆程度，同时保持甚至提升生成样本质量。理论分析基于随机特征神经网络模型 [Bonnaire et al., 2025]，表明一致性蒸馏可抑制与记忆相关的不稳定特征方向，保留稳定、可泛化的模式。结果表明，蒸馏不仅是加速工具，还可优化记忆-泛化权衡。
原文摘要 · Abstract (English)
Diffusion models are central to modern generative modeling, and understanding how they balance memorization and generalization is critical for reliable deployment. Recent work has shown that memorization in diffusion models is shaped by training dynamics, with generalization and memorization emerging at different stages of training. However, deployed diffusion models are often further distilled, introducing an additional training phase whose impact on memorization is not well understood. In this work, we analyze how distillation reshapes memorization behavior in diffusion models, taking consistency distillation as a representative framework. Empirically, we show that when applied to a teacher model that has memorized data, consistency distillation significantly reduces transferred memorization in the student while preserving, and sometimes improving, sample quality. To explain this behavior, we provide a theoretical analysis using a random feature neural network model [Bonnaire et al., 2025], showing that consistency distillation suppresses unstable feature directions associated with memorization while preserving stable, generalizable modes. Our findings suggest that distillation can serve not only as an acceleration tool, but also as a mechanism for improving the memorization-generalization trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。