评测图像自回归生成中的视觉分词器,发现连续变分自编码器表现更优
VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation
- 构建三任务基准测试视觉分词器:图像重建、细节保留、文本保留
- 连续VAE在结构与语义细节上显著优于离散分词器,后者易失真或丢纹理
- 适合关注自回归图像生成底层机制的研究者和开发者
自回归(AR)模型在图像生成中表现强劲,其关键组件是将连续像素映射为离散标记序列的视觉分词器(VT)。VT的质量决定了AR模型性能上限。然而,现有离散VT远落后于连续变分自编码器(VAEs),导致图像重建质量下降,细节与文本信息丢失。现有评估聚焦端到端生成效果,未能单独衡量VT性能。为此,我们提出VTBench,一个涵盖图像重建、细节保留、文本保留三类核心任务的综合性评估基准,覆盖多样评估场景。通过一系列指标系统评估前沿VTs,结果表明连续VAEs在保持空间结构和语义细节方面表现更优;而离散VT产生的表征劣化常引发重构失真、细粒度纹理丢失及文本与物体完整性失败。此外,我们在GPT-4o图像生成上开展实验,探讨其潜在的自回归特性,为视觉分词的作用提供新见解。我们公开发布基准与代码库,呼吁社区开发通用性强的开源视觉分词器。
原文摘要 · Abstract (English)
Autoregressive (AR) models have recently shown strong performance in image generation, where a critical component is the visual tokenizer (VT) that maps continuous pixel inputs to discrete token sequences. The quality of the VT largely defines the upper bound of AR model performance. However, current discrete VTs fall significantly behind continuous variational autoencoders (VAEs), leading to degraded image reconstructions and poor preservation of details and text. Existing benchmarks focus on end-to-end generation quality, without isolating VT performance. To address this gap, we introduce VTBench, a comprehensive benchmark that systematically evaluates VTs across three core tasks: Image Reconstruction, Detail Preservation, and Text Preservation, and covers a diverse range of evaluation scenarios. We systematically assess state-of-the-art VTs using a set of metrics to evaluate the quality of reconstructed images. Our findings reveal that continuous VAEs produce superior visual representations compared to discrete VTs, particularly in retaining spatial structure and semantic detail. In contrast, the degraded representations produced by discrete VTs often lead to distorted reconstructions, loss of fine-grained textures, and failures in preserving text and object integrity. Furthermore, we conduct experiments on GPT-4o image generation and discuss its potential AR nature, offering new insights into the role of visual tokenization. We release our benchmark and codebase publicly to support further research and call on the community to develop strong, general-purpose open-source VTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。