将视觉分词器扩展至30亿参数,显著提升图像生成与重建质量。
GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation
- 引入语义正则化,控制潜空间复杂度,解决缩放导致的生成退化问题。
- 在30亿参数规模下,重建与自回归生成性能均达当前最优水平。
- 适合追求高精度图像生成与大规模分词器设计的研究者参考。
在自回归图像生成中,视觉分词器将图像压缩为紧凑的离散潜在标记,使下游自回归模型可通过预测下一个标记实现高效训练。尽管扩大视觉分词器可提升图像重建质量,却常导致下游生成质量下降——这一挑战在现有文献中未得到充分解决。为此,我们提出GigaTok,首个在扩展视觉分词器时同时提升重建、生成与表征学习表现的方法。我们发现潜空间复杂度增长是重建与生成之间权衡的核心原因。为此,提出语义正则化,将分词器特征对齐预训练视觉编码器的语义一致特征,从而在缩放过程中抑制潜空间过复杂化,实现重建与下游生成的一致性提升。基于该正则化,探索三种关键扩展实践:(1) 使用一维分词器以增强可扩展性,(2) 在扩展编码器与解码器时优先扩展解码器,(3) 采用熵损失稳定百亿级分词器的训练。通过扩展至30亿参数,GigaTok在重建、下游自回归生成及表征质量方面均达到当前最优表现。
原文摘要 · Abstract (English)
In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality -- a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers:(1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to $\bf{3 \space billion}$ parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。