用软聚类聚合词元，实现高效图像压缩与生成。
SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer
- 用软类别后验将多个词元合并为一个，提升隐空间表示能力。
- 256x256图像仅需32个一维词元，生成速度最快提升18倍。
- 适合追求高效生成和训练的扩散模型研究者使用。
高效的图像词元化在高压缩比下仍具挑战性。本文提出SoftVQ-VAE，一种基于软类别后验的连续图像词元化方法，通过将多个码字聚合到每个隐词元中，显著提升隐空间的表征能力。应用于基于Transformer的架构时，该方法可将256x256和512x512图像压缩为仅32或64个一维词元。SoftVQ-VAE不仅实现稳定高质量重建，更重要的是，在多种基于去噪的生成模型中均取得领先且显著更快的生成性能：生成256x256图像时推理吞吐量最高提升18倍，512x512图像达55倍，同时在SiT-XL上保持1.78和2.21的竞争力FID分数。此外，训练效率提升2.3倍，而性能基本不变。其全可微设计与语义丰富的隐空间确保高效词元化不牺牲生成质量，为更高效的生成模型铺平道路。代码与模型已开源。
原文摘要 · Abstract (English)
Efficient image tokenization with high compression ratios remains a critical challenge for training generative models. We present SoftVQ-VAE, a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token, substantially increasing the representation capacity of the latent space. When applied to Transformer-based architectures, our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens. Not only does SoftVQ-VAE show consistent and high-quality reconstruction, more importantly, it also achieves state-of-the-art and significantly faster image generation results across different denoising-based generative models. Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images while achieving competitive FID scores of 1.78 and 2.21 for SiT-XL. It also improves the training efficiency of the generative models by reducing the number of training iterations by 2.3x while maintaining comparable performance. With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VAE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models. Code and model are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。