混合自回归与扩散模型，提升高分辨率图像生成质量与效率
MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation
- 分块处理图像，用自回归全局建模，扩散模型局部精修
- 在FFHQ-1024和ImageNet上，FID最高提升75%（计算受限下）
- 适合研究生成模型架构设计或追求高效高质图像生成的开发者
多模态生成近年来逐渐融合自回归（AR）与扩散模型，结合其互补优势：AR模型擅长捕捉长程依赖并生成连贯内容，扩散模型则在连续潜在空间中精细重构视觉细节。然而现有混合方法缺乏对模型容量分配的系统性指导。本文提出MADFormer，一种用于分析AR-扩散权衡的混合生成框架。该模型将图像划分为空间块，使用自回归层实现跨块的一次性全局条件建模，扩散层在每块内进行迭代局部优化。在FFHQ-1024和ImageNet上的受控实验表明：（1）块级划分显著提升高分辨率图像生成性能；（2）垂直混合AR与扩散层可实现更优的质量-效率平衡，在有限推理算力下使FID降低高达75%。研究成果为未来混合生成模型的设计提供了实用指导。
原文摘要 · Abstract (English)
Recent progress in multimodal generation has increasingly combined autoregressive (AR) and diffusion-based approaches, leveraging their complementary strengths: AR models capture long-range dependencies and produce fluent, context-aware outputs, while diffusion models operate in continuous latent spaces to refine high-fidelity visual details. However, existing hybrids often lack systematic guidance on how and why to allocate model capacity between these paradigms. In this work, we introduce MADFormer, a Mixed Autoregressive and Diffusion Transformer that serves as a testbed for analyzing AR-diffusion trade-offs. MADFormer partitions image generation into spatial blocks, using AR layers for one-pass global conditioning across blocks and diffusion layers for iterative local refinement within each block. Through controlled experiments on FFHQ-1024 and ImageNet, we identify two key insights: (1) block-wise partitioning significantly improves performance on high-resolution images, and (2) vertically mixing AR and diffusion layers yields better quality-efficiency balances--improving FID by up to 75% under constrained inference compute. Our findings offer practical design principles for future hybrid generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。