根据图像内容动态调整分块数量，更高效地表示复杂度不同的图像。
CAT: Content-Adaptive Image Tokenization
- 基于文本描述预测图像复杂度，自适应决定每张图的分块数量。
- 在相同计算量下，生成图像的FID分数优于固定分块方法。
- 适合需要高效图像生成与压缩的应用场景。
现有图像分块方法对所有图像采用固定数量的分块，忽略了图像复杂度的差异。为此，我们提出内容自适应分块器（CAT），根据图像内容动态调整表示容量，简单图像使用更少分块。设计了一种基于文本描述的评估系统，利用大语言模型（LLMs）预测内容复杂度，并确定最优压缩比，兼顾人类感知关键因素。在多种压缩比图像上训练后，CAT在图像重建任务中表现稳健。同时，利用其可变长度潜在表示，训练扩散变换器（DiTs）用于ImageNet图像生成。通过优化分块分配，CAT在相同浮点运算量下，相比固定比率基线提升FID分数，并使推理吞吐量提高18.5%。
原文摘要 · Abstract (English)
Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。