提出LIFT+PLACE框架，让小模型更好模仿大模型的去噪过程。
LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models
- 分步训练：先粗对齐再精细优化，提升学习稳定性。
- 在仅1.3万参数（教师1.6%）时仍稳定收敛，FID达15.73。
- 适用于图像/潜空间、U-Net/DiT等多类模型和任务。
我们发现，在扩散模型的知识蒸馏中，教师网络因容量过大而产生的复杂去噪过程，给学生模型的忠实模仿带来显著挑战。为此，我们提出一种由粗到精的蒸馏框架，包含基于线性拟合的蒸馏（LIFT）与分段局部自适应系数估计（PLACE）。LIFT将目标分解为“粗对齐”与“细调整”两阶段，学生先进行粗对齐训练，再进入硬性精调。PLACE进一步通过将输出按误差分组，提供局部自适应指导，以应对空间非均匀误差。实验表明，该方法在图像与潜空间、U-Net与DiT骨干网络、无条件与有条件生成任务、多种数据集上均有效，并可扩展至MMDiT（SD3）等流模型。在极端压缩下，学生模型仅130万参数（仅为教师的1.6%），传统知识蒸馏无法提供足够引导，导致训练不稳定，FID常升至50–200+；而本方法保持稳定收敛，最终获得FID 15.73。
原文摘要 · Abstract (English)
We demonstrate that in knowledge distillation for diffusion models, the teacher network's highly complex denoising process - stemming from its substantially larger capacity - poses a significant challenge for the student model to faithfully mimic. To address this problem, we propose a coarse-to-fine distillation framework with LInear FiTtingbased distillation (LIFT) and Piecewise Local Adaptive Coefficient Estimation (PLACE). First, LIFT decomposes the objective into a "coarse" alignment and a "fine" refinement. The student is then trained on coarse alignment before proceeding to hard refinement. Second, PLACE extends LIFT to address spatially non-uniform errors by partitioning outputs into error-based groups, providing locally adaptive guidance. Our experiments show that LIFT and PLACE is effective across diffusion spaces (image/latent), backbones (U-Net/DiT), tasks (unconditional/conditional), datasets, and even extends to flow-based models such as MMDiT (SD3). Furthermore, under extreme compression with a 1.3M-parameter student (only 1.6% of the teacher), conventional KD fails to provide sufficient guidance for stable training, with FID scores often degrading to 50-200+, but our method remains stably convergent and achieves an FID of 15.73.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。