用图像理解模型训练图像生成分词器，性能显著提升。
Image Understanding Makes for A Good Tokenizer for Image Generation
- 以图像理解模型为教师，训练具备语义能力的分词器。
- 在ImageNet-1k上实现4.10的FID，优于传统方法。
- 可直接将理解模型转为生成分词器，适合研究生成与理解融合者。
现代图像生成模型已展现出丰富的语义信息，可用于图像理解任务。然而，图像理解模型能否提升图像生成性能仍未知。本文采用基于分词的生成框架，通过特征重建目标训练分词器，即从预训练的图像理解编码器中蒸馏知识。实验表明，具备强理解能力的分词器在多种指标、数据集、任务和生成网络上均表现更优。其中，VQ-KD CLIP在ImageNet-1k上达到4.10的FID。可视化显示，其优势源于代码本中丰富的语义信息。此外，提出简单管道可直接将理解模型转化为生成分词器，效果卓越。这些发现或推动分词器研究，并重新思考理解与生成的关系。代码已开源。
原文摘要 · Abstract (English)
Abstract Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to project images into token sequences. Currently, pixel reconstruction (e.g., VQGAN) dominates the training objective for image tokenizers. In contrast, our approach adopts the feature reconstruction objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KD CLIP achieves $4.10$ FID on ImageNet-1k (IN-1k). Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vector_quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。