用视觉基础模型做图像生成的语义分词器，提升扩散模型速度与质量。
AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
- 将预训练视觉编码器对齐为扩散模型的语义分词器，三阶段优化。
- ImageNet上64轮即达gFID 1.90，收敛速度显著加快。
- 适合追求高效高质量图像生成的研究者与开发者。
本文提出将预训练视觉编码器对齐为潜在扩散模型的分词器，替代从头训练变分自编码器（VAE）的方法。传统VAE侧重低层细节，而本方法利用基础编码器丰富的语义结构。提出三阶段对齐策略：(1) 冻结编码器，训练适配器与解码器构建语义潜在空间；(2) 联合优化所有组件并引入语义保持损失，使编码器兼具感知细节与高层语义；(3) 优化解码器以提升重建质量。该对齐生成了富含语义的图像分词器，显著加速扩散模型收敛。在ImageNet 256×256上，仅64个训练周期即达gFID 1.90，且在有无分类器自由引导下均表现更优。扩展至LAION数据集，基于该分词器训练的文生图模型始终优于FLUX VAE与VA-VAE，相同训练步数下性能更佳。整体方法简单、可扩展，建立了连续分词器设计的语义基础范式。
原文摘要 · Abstract (English)
In this work, we propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation. Unlike training a variational autoencoder (VAE) from scratch, which primarily emphasizes low-level details, our approach leverages the rich semantic structure of foundation encoders. We introduce a three-stage alignment strategy called AlignTok: (1) freeze the encoder and train an adapter and a decoder to establish a semantic latent space; (2) jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture perceptual details while retaining high-level semantics; and (3) refine the decoder for improved reconstruction quality. This alignment yields semantically rich image tokenizers that benefit diffusion models. On ImageNet 256$\times$256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs, and improves generation both with and without classifier-free guidance. Scaling to LAION, text-to-image models trained with our tokenizer consistently outperforms FLUX VAE and VA-VAE under the same training steps. Overall, our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。