改进掩码自回归模型，实现高效高质图像生成
Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation
- 用双向注意力和2D RoPE重构架构，提升生成质量
- 1.4B参数模型仅需8步推理，FID达3.71，媲美主流AR模型
- 支持文本生成图像与实时语音转图像，适合高效生成场景
自回归（AR）模型在图像生成中取得显著进展，掩码自回归（MAR）模型因可并行解码而受到关注。但传统MAR模型性能落后于标准AR模型。本文通过评估多种图像分词器，优化出最优分词方案，并提出改进的双向LLaMA架构：以双向注意力替代因果注意力，引入2D RoPE，形成新型模型MaskGIL。该模型参数规模从111M扩展至1.4B，ImageNet 256x256基准上达到FID 3.71，与当前最先进AR模型相当，且仅需8次推理步骤，远少于标准AR模型的256步。此外，构建了775M参数的文本驱动版MaskGIL，支持多分辨率图像生成。该模型还可加速基于AR的生成流程，实现实时语音到图像转换。代码与模型已开源。
原文摘要 · Abstract (English)
AutoRegressive (AR) models have made notable progress in image generation, with Masked AutoRegressive (MAR) models gaining attention for their efficient parallel decoding. However, MAR models have traditionally underperformed when compared to standard AR models. This study refines the MAR architecture to improve image generation quality. We begin by evaluating various image tokenizers to identify the most effective one. Subsequently, we introduce an improved Bidirectional LLaMA architecture by replacing causal attention with bidirectional attention and incorporating 2D RoPE, which together form our advanced model, MaskGIL. Scaled from 111M to 1.4B parameters, MaskGIL achieves a FID score of 3.71, matching state-of-the-art AR models in the ImageNet 256x256 benchmark, while requiring only 8 inference steps compared to the 256 steps of AR models. Furthermore, we develop a text-driven MaskGIL model with 775M parameters for generating images from text at various resolutions. Beyond image generation, MaskGIL extends to accelerate AR-based generation and enable real-time speech-to-image conversion. Our codes and models are available at https://github.com/synbol/MaskGIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。