用专家路由实现多尺度视觉生成,提升质量与效率
MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

- 按尺度动态选择专家,解耦不同层级特征学习
- 在ImageNet上用一半训练轮次达更优FID,参数更少
- 适合追求高效高质量图像生成的研究者
视觉自回归建模(VAR)虽在图像生成中展现强大能力,但多尺度表示学习存在固有缺陷:低尺度侧重全局语义,高尺度关注细节,统一架构导致优化冲突。且因果自回归过程使早期语义错误会传播并严重损害输出。为此,本文提出一种尺度感知的令牌路由混合专家(MoE)架构,实现跨尺度自适应专家选择,促进表示解耦。同时,通过引入外部自监督特征增强早期语义建模,并设计适配VAR范式的残差特征聚合机制。大量实验表明,该方法显著提升训练效率与生成质量。在ImageNet 256×256基准上,相比密集基线模型,本方法仅需一半默认训练轮次与更小参数量,即可获得更优FID,训练成本仅小幅上升,且随训练轮次增加性能差距进一步扩大。
原文摘要 · Abstract (English)
Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256*256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。