调优扩散MoE模型架构，用更少参数实现更强性能。
Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
- 采用深邃风格专家模块与可变中间宽度设计
- 在相同激活参数下超越强基线模型性能
- 适合希望高效训练扩散模型的研究者
近期对扩散混合专家（Diffusion MoE）模型的研究主要聚焦于更复杂的路由机制。然而我们发现，底层的架构配置空间仍严重未被探索。受大型语言模型中MoE设计范式启发，我们识别出构建有效扩散MoE模型的关键架构因素——包括深邃风格专家模块、替代中间宽度、不同专家数量以及增强的注意力位置编码。系统的研究表明，精心调整这些配置对于释放扩散MoE模型全部潜力至关重要，其带来的增益往往超过仅靠路由创新所获。通过大量实验，我们提出了可高效应用于潜在空间与像素空间扩散框架的新架构，提供了一套实用且高效的训练方案，使扩散MoE模型在使用相等或更少激活参数的情况下超越强基线。所有代码与模型已公开：https://github.com/yhlleo/EfficientMoE。
原文摘要 · Abstract (English)
Recent efforts on Diffusion Mixture-of-Experts (MoE) models have primarily focused on developing more sophisticated routing mechanisms. However, we observe that the underlying architectural configuration space remains markedly under-explored. Inspired by the MoE design paradigms established in large language models (LLMs), we identify a set of crucial architectural factors for building effective Diffusion MoE models--including DeepSeek-style expert modules, alternative intermediate widths, varying expert counts, and enhanced attention positional encodings. Our systematic study reveals that carefully tuning these configurations is essential for unlocking the full potential of Diffusion MoE models, often yielding gains that exceed those achieved by routing innovations alone. Through extensive experiments, we present novel architectures that can be efficiently applied to both latent and pixel-space diffusion frameworks, which provide a practical and efficient training recipe that enables Diffusion MoE models to surpass strong baselines while using equal or fewer activated parameters. All code and models are publicly available at: https://github.com/yhlleo/EfficientMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。