arXiv:2510.11690cs.CVcs.LG2025-10被引 237

用更优的表示编码器替代传统VAE,提升扩散Transformer生成质量。

Diffusion Transformers with Representation Autoencoders

  • 用DINO等预训练编码器+自训练解码器构建新自编码器(RAE)
  • 在ImageNet上实现1.51 FID(无引导)和1.13 FID(有引导)
  • 无需额外对齐损失,收敛更快,适合高维潜在空间

潜变量生成建模中,预训练自编码器将像素映射到潜在空间用于扩散过程,已成为扩散Transformer(DiT)的标准策略;然而自编码器部分几乎未演进。多数DiT仍依赖原始VAE编码器,存在架构陈旧、潜在空间维度低、表征能力弱等问题,最终限制生成质量。本文探索以预训练表示编码器(如DINO、SigLIP、MAE)搭配训练解码器,构成新型表示自编码器(RAE)。该方法提供高质量重建与语义丰富的潜在空间,支持可扩展的Transformer架构。针对高维潜在空间带来的挑战,我们分析其成因,提出理论驱动的解决方案,并实证验证。采用轻量级宽头DDT结构的DiT变体,在ImageNet上取得优异结果:256×256无引导下FID为1.51,256×256与512×512有引导下均为1.13。RAE展现出明显优势,应成为扩散变压器训练的新默认方案。

原文摘要 · Abstract (English)

Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.

扩散模型自编码器图像生成Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。