将大模型高效设计引入扩散模型,实现生成质量与成本的更好平衡。
MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

- 采用路由专家、轻量路径等现代高效组件重构扩散Transformer架构。
- 在单机8卡训练下,比密集和中间稀疏基线收敛更快,FID更低。
- 适合关注生成效率与性能平衡的研究者或工业应用开发者。
现代大型语言模型通过容量增长与效率并进实现成功扩展,保持每令牌和部署成本可控。AIGC基础模型(AFMs),尤其是扩散-变压器骨干,开始采用稀疏专家,但近期工作大多仅增加总参数量和稀疏度,未引入使LLM扩展实用的效率机制,导致生成质量常难以兼顾训练与部署成本。这引出一个自然问题:能否将高效LLM扩展的架构原则以更均衡的方式适配到AFMs?我们提出ModernMOE(MMOE),对SiT风格扩散变压器进行现代化改造,系统性地引入路由专家、共享轻量专家、门控残差路由和注意力残差信息复用至AIGC生成中。不同于将MoE作为单一插件替换,MMOE研究不同现代专家组件在扩散变压器内组合时对收敛、效率和生成质量的影响。所有实验均在单台八卡H100节点上,以256批量大小运行40万步,符合可及的单机预算。在匹配训练与采样协议及该预算下,MMOE在每个记录检查点均达到更低的FID,即单位训练步数收敛更快,优于密集和中等稀疏基线;在稀疏变体中实现最佳质量-成本平衡。路由分析进一步显示:深度方向上专家专业化稳定,轻量路径使用充分,去噪过程中步骤间路由变化适中。结果表明,AFMs可通过引入经验证的效率设计,走与LLMs类似的均衡扩展路径,而非单纯增加参数总量与稀疏度。
原文摘要 · Abstract (English)
Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。