用冻结视觉大模型做图像分词器，生成更清晰、更快的图片。
Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
- 用冻结视觉大模型作编码器，配合自适应量化和语义重建目标。
- 在ImageNet上实现gFID 1.36，生成速度提升3倍，无需分类器自由引导。
- 适合追求高效高质图像生成的研究者与开发者。
本文提出一种新方法，直接基于冻结的视觉基础模型构建图像分词器，该方向尚属未充分探索。具体地，采用冻结的视觉基础模型作为分词器的编码器，并引入两个关键组件：(1) 区域自适应量化框架，降低预训练特征在规则2D网格上的冗余；(2) 语义重建目标，使分词器输出与基础模型表征对齐，保持语义保真度。基于此设计，所提出的图像分词器VFMTok在图像重建与生成质量上取得显著提升，同时增强标记效率。其进一步推动了自回归（AR）生成：在ImageNet基准上达到gFID 1.36，模型收敛速度提升三倍，并实现无需分类器自由引导的高保真类别条件合成。
原文摘要 · Abstract (English)
In this work, we present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model, which is a largely underexplored area. Specifically, we employ a frozen vision foundation model as the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features on regular 2D grids, and (2) a semantic reconstruction objective that aligns the tokenizer's outputs with the foundation model's representations to preserve semantic fidelity. Based on these designs, our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality, while also enhancing token efficiency. It further boosts autoregressive (AR) generation -- achieving a gFID of 1.36 on ImageNet benchmarks, while accelerating model convergence by three times, and enabling high-fidelity class-conditional synthesis without the need for classifier-free guidance (CFG). The code is available at https://github.com/CVMI-Lab/VFMTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。