用频谱顺序生成图像，实现高效且因果的自回归视觉生成。
SpectralAR: Spectral Autoregressive Visual Generation
- 将图像转为从低频到高频的有序频谱令牌序列，确保生成因果性。
- 仅用64个令牌和310M参数，图像重建达到3.02 gFID。
- 适合追求高效率与理论一致性的自回归图像生成研究者。
自回归视觉生成因可扩展性和与其他模态的兼容性，相比扩散模型受到越来越多关注。现有方法通常将图像划分为空间补丁进行自回归生成，但图像补丁本质上是并行的，与自回归建模的因果性相矛盾。为此，我们提出一种从频谱视角实现因果性的自回归视觉生成框架SpectralAR。具体地，通过嵌套频谱标记化（Nested Spectral Tokenization）将图像转换为从低频到高频的有序频谱令牌序列，并以粗到细的方式进行自回归生成。该方法在保留图像不同细节层级的同时，实现了序列因果性与令牌效率，无需额外复杂设计。我们在ImageNet-1K上进行了广泛实验，结果显示SpectralAR仅使用64个令牌和310M参数，即达到3.02 gFID的图像重建性能。
原文摘要 · Abstract (English)
Autoregressive visual generation has garnered increasing attention due to its scalability and compatibility with other modalities compared with diffusion models. Most existing methods construct visual sequences as spatial patches for autoregressive generation. However, image patches are inherently parallel, contradicting the causal nature of autoregressive modeling. To address this, we propose a Spectral AutoRegressive (SpectralAR) visual generation framework, which realizes causality for visual sequences from the spectral perspective. Specifically, we first transform an image into ordered spectral tokens with Nested Spectral Tokenization, representing lower to higher frequency components. We then perform autoregressive generation in a coarse-to-fine manner with the sequences of spectral tokens. By considering different levels of detail in images, our SpectralAR achieves both sequence causality and token efficiency without bells and whistles. We conduct extensive experiments on ImageNet-1K for image reconstruction and autoregressive generation, and SpectralAR achieves 3.02 gFID with only 64 tokens and 310M parameters. Project page: https://huang-yh.github.io/spectralar/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。