arXiv:2412.05796cs.CVcs.AI2024-12CVPR被引 37

用文字指导图像分块,让生成更高效清晰。

Language-Guided Image Tokenization for Generation

  • 用文本描述引导图像分块,生成更紧凑的语义表示。
  • 在ImageNet上压缩率更高,重建质量提升29.2%~48.1%。
  • 适合追求高速生成与高画质的图像生成研究者。

图像分块是将原始像素转换为低维潜在表示的关键步骤,对高效图像生成至关重要。然而主流方法压缩率有限,导致高分辨率生成计算成本高。本文提出语言引导的图像分块方法TexTok,通过文本描述条件化分块过程,获得更紧凑的高层语义表征。该方法简化语义学习,将更多容量用于捕捉细节,从而提升重建质量和压缩率。相比无文本条件的基线,TexTok在ImageNet-256和-512上分别实现平均重建FID提升29.2%和48.1%,生成FID平均提升16.3%和34.3%。仅用32个令牌替换DiT中的分块器,即可实现93.5倍推理加速,同时超越原DiT模型。TexTok结合普通DiT生成器,在ImageNet-256和-512上分别取得1.46和1.62的SOTA FID分数。此外,在文本到图像生成任务中,可直接利用现成文本描述,表现优异。

原文摘要 · Abstract (English)

Image tokenization, the process of transforming raw image pixels into a compact low-dimensional latent representation, has proven crucial for scalable and efficient image generation. However, mainstream image tokenization methods generally have limited compression rates, making high-resolution image generation computationally expensive. To address this challenge, we propose to leverage language for efficient image tokenization, and we call our method Text-Conditioned Image Tokenization (TexTok). TexTok is a simple yet effective tokenization framework that leverages language to provide a compact, high-level semantic representation. By conditioning the tokenization process on descriptive text captions, TexTok simplifies semantic learning, allowing more learning capacity and token space to be allocated to capture fine-grained visual details, leading to enhanced reconstruction quality and higher compression rates. Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29.2% and 48.1% on ImageNet-256 and -512 benchmarks respectively, across varying numbers of tokens. These tokenization improvements consistently translate to 16.3% and 34.3% average improvements in generation FID. By simply replacing the tokenizer in Diffusion Transformer (DiT) with TexTok, our system can achieve a 93.5x inference speedup while still outperforming the original DiT using only 32 tokens on ImageNet-512. TexTok with a vanilla DiT generator achieves state-of-the-art FID scores of 1.46 and 1.62 on ImageNet-256 and -512 respectively. Furthermore, we demonstrate TexTok's superiority on the text-to-image generation task, effectively utilizing the off-the-shelf text captions in tokenization. Project page is at: https://kaiwenzha.github.io/textok/.

图像生成文本引导分块优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。