拆解再合并：让扩散模型分阶段优化，生成更清晰。
Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning
- 先为不同去噪步骤训练独立模型，再融合成一个
- 在6个数据集上显著提升图像生成质量
- 适合想改进扩散模型性能的研究者和开发者
扩散模型通过学习逆转每一步噪声添加的过程进行训练。通常，模型参数在所有时间步共享以提高训练效率。然而，由于每个时间步的去噪任务不同，各步计算出的梯度可能冲突，从而降低整体生成性能。为此，本文提出一种解耦-合并（DeMe）框架：从预训练模型出发，为特定时间步微调独立模型，并在微调阶段引入多项改进技术，促进知识共享的同时最小化时间步间的训练干扰。微调完成后，这些独立模型可在参数空间中合并为单一模型，实现高效实用的推理。实验表明，在6个基准测试中均有显著提升，包括Stable Diffusion在COCO30K、ImageNet1K、PartiPrompts，以及DDPM在LSUN Church、LSUN Bedroom和CIFAR10上的表现。代码已开源。
原文摘要 · Abstract (English)
Diffusion models are trained by learning a sequence of models that reverse each step of noise corruption. Typically, the model parameters are fully shared across multiple timesteps to enhance training efficiency. However, since the denoising tasks differ at each timestep, the gradients computed at different timesteps may conflict, potentially degrading the overall performance of image generation. To solve this issue, this work proposes a \textbf{De}couple-then-\textbf{Me}rge (\textbf{DeMe}) framework, which begins with a pretrained model and finetunes separate models tailored to specific timesteps. We introduce several improved techniques during the finetuning stage to promote effective knowledge sharing while minimizing training interference across timesteps. Finally, after finetuning, these separate models can be merged into a single model in the parameter space, ensuring efficient and practical inference. Experimental results show significant generation quality improvements upon 6 benchmarks including Stable Diffusion on COCO30K, ImageNet1K, PartiPrompts, and DDPM on LSUN Church, LSUN Bedroom, and CIFAR10. Code is available at \href{https://github.com/MqLeet/DeMe}{GitHub}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。