动态选择令牌，让扩散模型更高效生成图像
DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
- 根据噪声水平动态分配计算资源，提升处理效率
- 在ImageNet上超越现有模型，参数激活量仅1倍
- 适用于图文生成等复杂任务，通用性强
扩散模型在图像生成任务中表现卓越，但其性能常受限于对不同条件和噪声水平下输入的统一处理。为突破这一限制，我们提出一种新方法，利用扩散过程的内在异质性。DiffMoE引入批量级全局令牌池，使专家在训练中可访问全局令牌分布，促进专业化行为。为充分释放扩散过程潜力，该方法还集成容量预测器，依据噪声水平和样本复杂度动态分配计算资源。全面评估显示，DiffMoE在ImageNet基准上达到当前最优性能，显著优于参数激活量达3倍的密集架构及现有MoE方法，同时保持1倍激活参数。该方法有效性不仅体现在类别条件生成，还扩展至更具挑战性的文本到图像生成任务，展现出广泛适用性。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable success in various image generation tasks, but their performance is often limited by the uniform processing of inputs across varying conditions and noise levels. To address this limitation, we propose a novel approach that leverages the inherent heterogeneity of the diffusion process. Our method, DiffMoE, introduces a batch-level global token pool that enables experts to access global token distributions during training, promoting specialized expert behavior. To unleash the full potential of the diffusion process, DiffMoE incorporates a capacity predictor that dynamically allocates computational resources based on noise levels and sample complexity. Through comprehensive evaluation, DiffMoE achieves state-of-the-art performance among diffusion models on ImageNet benchmark, substantially outperforming both dense architectures with 3x activated parameters and existing MoE approaches while maintaining 1x activated parameters. The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation, demonstrating its broad applicability across different diffusion model applications. Project Page: https://shiml20.github.io/DiffMoE/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。