用视觉大模型直接做通用图像分词器,提升生成质量与效率。
Vision Foundation Models as Generalist Tokenizers for Image Generation

- 基于冻结的视觉大模型,设计自适应区域量化与语义重建机制。
- 离散生成加速3倍,图像质量达gFID 1.36;连续生成达gFID 1.25。
- 无需分类器自由引导,可高效实现高保真条件生成,适合图像生成研究者。
本文探索了在冻结的视觉基础模型(VFM)之上构建通用图像分词器的新方向。我们提出VFMTok,利用冻结的VFM作为编码器,引入两项关键创新:(1) 区域自适应量化框架,消除标准2D网格特征中的空间冗余;(2) 语义重建目标,使解码输出与VFM表示对齐,保持语义一致性。基于此设计,VFMTok可在离散与连续潜在空间中无缝运行。在离散自回归生成中,其收敛速度提升3倍,ImageNet类条件合成上达到1.36的gFID;在连续空间中,与去噪模型结合后取得1.25的优异gFID。由于潜在空间蕴含丰富空间语义,该方法无需分类器自由引导即可实现高保真类条件生成,显著加快推理速度。进一步分析表明,VFM预训练时采用的自监督学习目标(全局对比学习与潜变量掩码图像建模)决定其作为分词器的有效性,为未来分词器设计提供重要指导。
原文摘要 · Abstract (English)
In this work, we explore the largely unexplored direction of building a generalist image tokenizer directly on top of a frozen vision foundation model (VFM). To build this tokenizer, we utilize a frozen VFM as the encoder and introduce two key innovations: (1) a region-adaptive quantization framework to eliminate spatial redundancy in standard 2D grid features, and (2) a semantic reconstruction objective that aligns the decoded outputs with the VFM's representations to preserve semantic fidelity. Grounded in these designs, we propose VFMTok, a generalist visual tokenizer capable of operating seamlessly in both discrete and continuous latent spaces. VFMTok achieves substantial improvements in synthesis quality while drastically enhancing token efficiency. For discrete autoregressive (AR) generation, it accelerates model convergence by \textbf{3 times} and achieves a state-of-the-art gFID of \textbf{1.36} on ImageNet class-conditional synthesis. Similarly, for continuous-space generation, integrating VFMTok with a denoising model yields an exceptional gFID of \textbf{1.25}. Furthermore, because the latent space inherently captures rich spatial semantics, VFMTok enables high-fidelity class-conditional synthesis without classifier-free guidance (\textbf{w/o CFG}) across both generative paradigms, significantly accelerating inference speed. Beyond these remarkable empirical results, we systematically investigate the underlying mechanisms of our approach. We discover that the specific self-supervised learning objectives utilized during VFM pre-training dictate its effectiveness as a tokenizer. Specifically, a VFM jointly optimized with global contrastive learning and latent masked image modeling provides the optimal representations for image tokenization. These insights establish a strong foundation and offer valuable guidance for the design of future image tokenizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。