将图像生成的尺度预测解耦,提升质量与速度。
M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation
- 分离尺度内与尺度间建模,优化计算效率。
- 1.5B模型参数更少但性能超2B的VAR-d30。
- 适合追求高效高质图像生成的研究与应用。
近期计算机视觉领域提出一种名为VAR的新自回归图像生成范式,将图像生成重构为从粗到细的逐尺度预测。本文表明,该尺度自回归框架可有效解耦为:捕捉尺度内局部空间依赖的「尺度内建模」,以及逐步建模跨尺度关系的「尺度间建模」。这种解耦结构使模型重建更高效。具体而言,对生成高保真图像至关重要的尺度内建模保留原始双向自注意力设计;而语义连接多尺度、计算开销大的尺度间建模,则采用线性复杂度机制(如Mamba)显著降低计算负担。我们提出的新框架命名为M-VAR。大量实验表明,该方法在图像质量与生成速度上均优于现有模型。例如,我们的1.5B参数模型在参数更少、推理更快的情况下,超越了最大的VAR-d30-2B模型。此外,最大规模模型M-VAR-d32在ImageNet 256×256上取得1.78的FID,分别优于前代自回归模型LlamaGen/VAR(0.4/0.19)和主流扩散模型LDM/DiT(1.82/0.49)。代码已开源。
原文摘要 · Abstract (English)
There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales. This decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling -- crucial for generating high-fidelity images -- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. We term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d30-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively. Code is avaiable at \url{https://github.com/OliverRensu/MVAR}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。