统一视觉生成与理解的分词器，提升性能并支持端到端生成。
UniTok: A Unified Tokenizer for Visual Generation and Understanding
- 创新多码本量化机制扩大离散编码空间容量
- 图像生成与理解任务分别达0.38 rFID和78.6%零样本准确率
- 兼容多模态大模型，支持无条件生成且不损失理解能力
视觉生成与理解模型通常依赖不同的分词器处理图像，难以在单一框架中统一。现有研究尝试通过联合训练VQVAE（用于自回归生成）与CLIP（用于理解）构建统一分词器，但直接结合训练目标常引发严重损失冲突。本文指出，重建与语义监督本身并不矛盾，根本瓶颈在于离散编码空间表征能力有限。基于此，提出UniTok，采用新型多码本量化机制有效扩展词汇量与瓶颈维度。实验表明，UniTok在ImageNet上达到0.38 rFID和78.6%零样本准确率新纪录。同时可无缝集成至多模态大模型，实现原生视觉生成能力而不影响理解性能。此外，其支持无条件生成，在ImageNet 256×256基准上将gFID从14.6降至2.5。
原文摘要 · Abstract (English)
Visual generative and understanding models typically rely on distinct tokenizers to process images, presenting a key challenge for unifying them within a single framework. Recent studies attempt to address this by connecting the training of VQVAE (for autoregressive generation) and CLIP (for understanding) to build a unified tokenizer. However, directly combining these training objectives has been observed to cause severe loss conflicts. In this paper, we show that reconstruction and semantic supervision do not inherently conflict. Instead, the underlying bottleneck stems from limited representational capacity of discrete token space. Building on these insights, we introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism that effectively scales up the vocabulary size and bottleneck dimension. In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6% zero-shot accuracy on ImageNet. Besides, UniTok can be seamlessly integrated into MLLMs to unlock native visual generation capability, without compromising the understanding performance. Additionally, we show that UniTok favors cfg-free generation, reducing gFID from 14.6 to 2.5 on ImageNet 256$\times$256 benchmark. GitHub: https://github.com/FoundationVision/UniTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。