用频谱渐进方式生成图像，提升自回归模型效率与质量
Frequency Autoregressive Image Generation with Continuous Tokens
- 以频谱依赖为回归方向，从低频到高频逐步生成图像
- 在ImageNet上实现高质量图像生成，训练与推理更高效
- 适合关注图像生成新范式的研究者与工程师
图像自回归生成模型通常采用向量量化与光栅扫描的两阶段范式，受语言建模成功启发。然而，由于模态差距大，需从分词器格式和回归方向两个角度重新审视。本文提出频谱渐进自回归（FAR）范式，并使用连续分词器实现。我们发现频谱依赖是理想的回归方向：高频分量基于低频分量逐步构建完整图像。该设计自然满足自回归因果性要求，同时保持图像数据的空间局部性。此外，深入探索了FAR与连续分词器的结合，引入一系列技术解决优化难题，提升训练与推理效率。在ImageNet数据集上的实验验证了FAR的有效性，并在文本到图像生成任务中展现出潜力。
原文摘要 · Abstract (English)
Autoregressive (AR) models for image generation typically adopt a two-stage paradigm of vector quantization and raster-scan ``next-token prediction", inspired by its great success in language modeling. However, due to the huge modality gap, image autoregressive models may require a systematic reevaluation from two perspectives: tokenizer format and regression direction. In this paper, we introduce the frequency progressive autoregressive (\textbf{FAR}) paradigm and instantiate FAR with the continuous tokenizer. Specifically, we identify spectral dependency as the desirable regression direction for FAR, wherein higher-frequency components build upon the lower one to progressively construct a complete image. This design seamlessly fits the causality requirement for autoregressive models and preserves the unique spatial locality of image data. Besides, we delve into the integration of FAR and the continuous tokenizer, introducing a series of techniques to address optimization challenges and improve the efficiency of training and inference processes. We demonstrate the efficacy of FAR through comprehensive experiments on the ImageNet dataset and verify its potential on text-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。