通过分层语义细节建模,显著提升自回归图像生成质量与效率。
IAR2: Improving Autoregressive Visual Generation with Semantic-Detail Associated Token Prediction
- 构建语义与细节双代码本,实现粗到细的分层生成。
- 在ImageNet上达到FID 1.50,刷新自回归图像生成纪录。
- 适合追求高质量图像生成与计算效率的研究者使用。
自回归模型在视觉内容生成中表现强大,但常忽略视觉数据的内在结构特性。先前工作IAR通过基于嵌入相似性的视觉代码本重组提升了生成鲁棒性,但受限于预训练代码本的刚性及硬性均匀聚类的不准确性。为此,本文提出IAR2,一种先进的自回归框架,支持分层语义-细节合成过程。核心是新型语义-细节关联双代码本,将图像表示解耦为全局语义代码本与细粒度细节代码本,量化能力从线性扩展至多项式级别,显著增强表达力。为此,提出语义-细节自回归预测机制与局部上下文增强的自回归头,分步预测:先语义令牌,再细节令牌,并利用局部上下文窗口提升空间一致性。针对条件生成,引入渐进式注意力引导自适应CFG机制,根据条件相关性与生成序列中的时间位置动态调节每令牌的引导强度,提升条件对齐性而不损失真实感。大量实验表明,IAR2在自回归图像生成上达到新SOTA,ImageNet上FID达1.50。模型不仅性能超越以往方法,且计算效率更优,验证了结构化粗到细生成策略的有效性。
原文摘要 · Abstract (English)
Autoregressive models have emerged as a powerful paradigm for visual content creation, but often overlook the intrinsic structural properties of visual data. Our prior work, IAR, initiated a direction to address this by reorganizing the visual codebook based on embedding similarity, thereby improving generation robustness. However, it is constrained by the rigidity of pre-trained codebooks and the inaccuracies of hard, uniform clustering. To overcome these limitations, we propose IAR2, an advanced autoregressive framework that enables a hierarchical semantic-detail synthesis process. At the core of IAR2 is a novel Semantic-Detail Associated Dual Codebook, which decouples image representations into a semantic codebook for global semantic information and a detail codebook for fine-grained refinements. It expands the quantization capacity from a linear to a polynomial scale, significantly enhancing expressiveness. To accommodate this dual representation, we propose a Semantic-Detail Autoregressive Prediction scheme coupled with a Local-Context Enhanced Autoregressive Head, which performs hierarchical prediction-first the semantic token, then the detail token-while leveraging a local context window to enhance spatial coherence. Furthermore, for conditional generation, we introduce a Progressive Attention-Guided Adaptive CFG mechanism that dynamically modulates the guidance scale for each token based on its relevance to the condition and its temporal position in the generation sequence, improving conditional alignment without sacrificing realism. Extensive experiments demonstrate that IAR2 sets a new state-of-the-art for autoregressive image generation, achieving a FID of 1.50 on ImageNet. Our model not only surpasses previous methods in performance but also demonstrates superior computational efficiency, highlighting the effectiveness of our structured, coarse-to-fine generation strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。