用掩码自编码器构建高效图像生成的潜在表征空间
Masked Autoencoders Are Effective Tokenizers for Diffusion Models
- 用掩码建模训练自编码器，学习语义丰富的潜在表示
- 仅用128个令牌实现ImageNet上顶尖生成质量，gFID达1.69
- 适合追求高效生成与快速训练的扩散模型研究者
近年来，潜在扩散模型在高分辨率图像合成中表现出色。然而，用于扩散模型的潜在空间表征特性仍缺乏深入探索。理论与实证分析表明，生成质量的提升与潜在分布结构密切相关，如更少的高斯混合模式和更强的区分性特征。受此启发，我们提出MAETok，一种基于掩码建模的自编码器，能在保持重建保真度的同时学习语义丰富的潜在空间。大量实验验证了我们的分析：变分形式的自编码器并非必要，仅通过自编码器即可获得优秀的潜在空间，实现仅用128个令牌在ImageNet上生成512x512图像的领先性能。MAETok显著提升效率：训练速度加快76倍，推理吞吐量提高31倍，且达到gFID 1.69。结果表明，潜在空间的结构而非变分约束才是扩散模型有效性的关键。代码与训练模型已公开。
原文摘要 · Abstract (English)
Recent advances in latent diffusion models have demonstrated their effectiveness for high-resolution image synthesis. However, the properties of the latent space from tokenizer for better learning and generation of diffusion models remain under-explored. Theoretically and empirically, we find that improved generation quality is closely tied to the latent distributions with better structure, such as the ones with fewer Gaussian Mixture modes and more discriminative features. Motivated by these insights, we propose MAETok, an autoencoder (AE) leveraging mask modeling to learn semantically rich latent space while maintaining reconstruction fidelity. Extensive experiments validate our analysis, demonstrating that the variational form of autoencoders is not necessary, and a discriminative latent space from AE alone enables state-of-the-art performance on ImageNet generation using only 128 tokens. MAETok achieves significant practical improvements, enabling a gFID of 1.69 with 76x faster training and 31x higher inference throughput for 512x512 generation. Our findings show that the structure of the latent space, rather than variational constraints, is crucial for effective diffusion models. Code and trained models are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。