arXiv:2501.09755cs.CVcs.AI2025-01ICML被引 31

探索视觉分词器的扩展规律,发现解码器扩展更利于生成性能。

Learnings from Scaling Visual Tokenizers for Reconstruction and Generation

  • 用ViT替代卷积,构建可扩展的视觉分词器ViTok
  • 解码器扩展显著提升重建效果,对生成有混合影响
  • 轻量设计在多项任务上超越现有模型,计算量更低

通过自编码实现视觉分词,将像素压缩至潜在空间,赋能当前顶尖的图像与视频生成模型。尽管基于Transformer的生成器持续扩展是近年进展的核心,但分词器本身却很少被放大,其设计选择如何影响重建和下游生成性能仍不明确。本文旨在填补这一空白,提出使用增强型视觉变换器架构(ViTok)替代传统卷积主干,训练于远超ImageNet-1K规模的图像与视频数据集,打破分词器扩展的数据限制。研究发现:扩大自编码器瓶颈虽与重建高度相关,但与生成性能关系复杂;单独扩展编码器对重建和生成均无显著提升,而扩展解码器虽能改善重建,生成收益则不一致。基于此,设计出轻量级分词器ViTok,其在ImageNet-1K和COCO(256p、512p)重建任务上表现媲美当前最优,且在UCF-101 16帧128p视频重建中优于现有模型,同时减少2-5倍计算量(FLOPs)。集成到扩散变换器后,ViTok在ImageNet-1K图像生成中表现竞争力,并在UCF-101类别条件视频生成上创下新基准。

原文摘要 · Abstract (English)

Visual tokenization via auto-encoding empowers state-of-the-art image and video generative models by compressing pixels into a latent space. Although scaling Transformer-based generators has been central to recent advances, the tokenizer component itself is rarely scaled, leaving open questions about how auto-encoder design choices influence both its objective of reconstruction and downstream generative performance. Our work aims to conduct an exploration of scaling in auto-encoders to fill in this blank. To facilitate this exploration, we replace the typical convolutional backbone with an enhanced Vision Transformer architecture for Tokenization (ViTok). We train ViTok on large-scale image and video datasets far exceeding ImageNet-1K, removing data constraints on tokenizer scaling. We first study how scaling the auto-encoder bottleneck affects both reconstruction and generation -- and find that while it is highly correlated with reconstruction, its relationship with generation is more complex. We next explored the effect of separately scaling the auto-encoders' encoder and decoder on reconstruction and generation performance. Crucially, we find that scaling the encoder yields minimal gains for either reconstruction or generation, while scaling the decoder boosts reconstruction but the benefits for generation are mixed. Building on our exploration, we design ViTok as a lightweight auto-encoder that achieves competitive performance with state-of-the-art auto-encoders on ImageNet-1K and COCO reconstruction tasks (256p and 512p) while outperforming existing auto-encoders on 16-frame 128p video reconstruction for UCF-101, all with 2-5x fewer FLOPs. When integrated with Diffusion Transformers, ViTok demonstrates competitive performance on image generation for ImageNet-1K and sets new state-of-the-art benchmarks for class-conditional video generation on UCF-101.

视觉分词生成模型Transformer轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。