通过动态屏蔽部分参数，提升扩散模型生成质量。
Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability
- 按时间步和样本动态屏蔽U-Net无关参数，专注关键信息。
- 零样本推理下在COCO数据集上达到最优FID分数。
- 无需额外参数，适合对生成质量要求高的场景。
扩散模型在早期阶段关注图像基本结构，后期才生成细节特征。传统网络（如ResNet或GAN）在不同层捕捉语义信息，而扩散模型却要求同一层同时学习结构与纹理，造成学习负担。为此，我们研究了U-Net参数对去噪过程的贡献，发现合理置零某些参数（包括大参数）能显著提升生成质量。基于此提出简单有效的「MaskUNet」方法，仅通过激活有效参数即可增强生成能力。提供两种优化策略：训练型与免训练型，包含定制网络与优化函数。在COCO数据集的零样本推理中，MaskUNet取得最优FID分数，并在下游任务评估中表现优异。
原文摘要 · Abstract (English)
The diffusion models, in early stages focus on constructing basic image structures, while the refined details, including local features and textures, are generated in later stages. Thus the same network layers are forced to learn both structural and textural information simultaneously, significantly differing from the traditional deep learning architectures (e.g., ResNet or GANs) which captures or generates the image semantic information at different layers. This difference inspires us to explore the time-wise diffusion models. We initially investigate the key contributions of the U-Net parameters to the denoising process and identify that properly zeroing out certain parameters (including large parameters) contributes to denoising, substantially improving the generation quality on the fly. Capitalizing on this discovery, we propose a simple yet effective method-termed ``MaskUNet''- that enhances generation quality with negligible parameter numbers. Our method fully leverages timestep- and sample-dependent effective U-Net parameters. To optimize MaskUNet, we offer two fine-tuning strategies: a training-based approach and a training-free approach, including tailored networks and optimization functions. In zero-shot inference on the COCO dataset, MaskUNet achieves the best FID score and further demonstrates its effectiveness in downstream task evaluations. Project page: https://gudaochangsheng.github.io/MaskUnet-Page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。