用专家混合策略加速扩散模型采样，自动分配计算资源。
Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection
- 将扩散模型按时间步分段，训练专用专家网络。
- 在多个数据集上实现采样速度提升，且生成质量保持不变。
- 适合需要高效推理的图像生成应用开发者。
扩散概率模型能生成高质量样本，但其采样过程需大量去噪步骤，导致速度慢且计算开销大。本文提出通过剪枝预训练扩散模型，构建高效专家混合体来降低采样成本。首先，研究不同去噪时间步之间的相似性，发现即使在不同数据集间也存在自然聚类现象，暗示可为每个时间区间配备独立的专用模型作为‘专家’。因此，我们对每个时间区间分别微调预训练模型，采用弹性深度与宽度结构，获得针对特定去噪区间的专家。为优化专家间的资源分配，引入专家路由代理（Expert Routing Agent），学习选择合适的网络配置，实现计算预算在专家间的端到端优化，无需手动设定启发式规则。最后，基于选定配置，对剪枝后的专家进行微调，形成最终的高效专家混合模型。我们在潜空间扩散模型架构下，在LSUN-Church、LSUN-Beds、FFHQ和ImageNet等多个数据集上验证了该方法的有效性。
原文摘要 · Abstract (English)
Diffusion probabilistic models can generate high-quality samples. Yet, their sampling process requires numerous denoising steps, making it slow and computationally intensive. We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. First, we study the similarities between pairs of denoising timesteps, observing a natural clustering, even across different datasets. This suggests that rather than having a single model for all time steps, separate models can serve as ``experts'' for their respective time intervals. As such, we separately fine-tune the pretrained model on each interval, with elastic dimensions in depth and width, to obtain experts specialized in their corresponding denoising interval. To optimize the resource usage between experts, we introduce our Expert Routing Agent, which learns to select a set of proper network configurations. By doing so, our method can allocate the computing budget between the experts in an end-to-end manner without requiring manual heuristics. Finally, with a selected configuration, we fine-tune our pruned experts to obtain our mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets, LSUN-Church, LSUN-Beds, FFHQ, and ImageNet, on the Latent Diffusion Model architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。