用256个令牌实现1024像素图像高保真重建，压缩比达16倍。
Layton: Latent Consistency Tokenizer for 1024-pixel Image Reconstruction and Generation by 256 Tokens
- 将视觉令牌映射到预训练扩散模型的紧凑隐空间，实现高效编码。
- 1024×1024图像重建FID达10.8，显著优于现有方法。
- 适用于高分辨率图像生成与文本到图像建模，适合追求效率与质量的研究者。
图像标记化在视觉生成与多模态建模中取得显著进展，尤其与自回归模型结合时表现突出。然而，现有方法难以兼顾效率与保真度：高分辨率图像重建要么需大量标记，要么通过压缩损失关键细节。为此，我们提出隐一致性标记器（Layton），将离散视觉标记与预训练隐扩散模型（LDM）的紧凑隐空间相连接，仅用256个标记即可表示1024×1024图像，相比VQGAN压缩率达16倍。Layton包含一个Transformer编码器、量化码本和隐一致性解码器。直接使用LDM作为解码器会导致色彩与亮度偏差，因此我们将其转化为隐一致性解码器，将多步采样减少至1-2步，实现像素级监督。实验表明，Layton在高保真重建上表现优异，在MSCOCO-2017 5K基准上达到10.8的重建弗雷切特初始距离（FID）。我们还将Layton扩展为文本到图像生成模型LaytonGen，采用自回归机制，其在GenEval基准上得分0.73，超越当前最优方法。
原文摘要 · Abstract (English)
Image tokenization has significantly advanced visual generation and multimodal modeling, particularly when paired with autoregressive models. However, current methods face challenges in balancing efficiency and fidelity: high-resolution image reconstruction either requires an excessive number of tokens or compromises critical details through token reduction. To resolve this, we propose Latent Consistency Tokenizer (Layton) that bridges discrete visual tokens with the compact latent space of pre-trained Latent Diffusion Models (LDMs), enabling efficient representation of 1024x1024 images using only 256 tokens-a 16 times compression over VQGAN. Layton integrates a transformer encoder, a quantized codebook, and a latent consistency decoder. Direct application of LDM as the decoder results in color and brightness discrepancies. Thus, we convert it to latent consistency decoder, reducing multi-step sampling to 1-2 steps for direct pixel-level supervision. Experiments demonstrate Layton's superiority in high-fidelity reconstruction, with 10.8 reconstruction Frechet Inception Distance on MSCOCO-2017 5K benchmark for 1024x1024 image reconstruction. We also extend Layton to a text-to-image generation model, LaytonGen, working in autoregression. It achieves 0.73 score on GenEval benchmark, surpassing current state-of-the-art methods. Project homepage: https://github.com/OPPO-Mente-Lab/Layton
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。