用改进的表示自编码器,让图像生成更快更准。
Improved Baselines with Representation Autoencoders

- 融合多层编码特征提升重建质量,无需微调或特殊数据
- 新方法使训练速度提升10倍,ImageNet-256上达最优gFID 1.06
- 可免费提供无分类器引导,适合高效图像生成研究者
表示自编码器(RAE)用预训练视觉编码器替代传统VAE。本文系统研究多个设计选择,发现三项关键洞见:首先,将表示定义为最后k个编码层的叠加而非仅最后一层,显著提升重建性能,且无需编码器微调或专用数据(如文本、人脸);其次,通过大规模实证分析发现,RAE与表示对齐(REPA)机制互补,同一表示可同时作为编码器和中间扩散层的目标;最后,原RAE在无分类器引导(CFG)下表现不佳,需训练第二个弱模型实现自动引导(AG),而本文证明REPA本质上是RAE潜空间中的x-prediction,仅重参数化DiT输出即可免费提供引导。由此提出的RAEv2实现超过10倍的收敛速度提升,在ImageNet-256上仅用80轮即达状态最优的gFID 1.06;在FDr6上以80轮达成2.17的最优结果(此前最佳3.26,耗时800轮),无需后处理。提出EPFID@k(达到无引导gFID<k所需的轮数)衡量训练效率,RAEv2达EPFID@2为35轮,原RAE为177轮。该方法在文生图与导航世界模型中均验证有效。代码已公开于https://raev2.github.io。
原文摘要 · Abstract (English)
Representation Autoencoders (RAE) replace traditional VAE with pretrained vision encoders. In this paper, we systematically investigate several design choices and find three insights which simplify and improve RAE. First, we study a generalized formulation where the representation is defined as sum of the last k encoder layers rather than solely the final layer. This simple change greatly improves reconstruction without encoder finetuning or specialized data (e.g., text, faces). Second, we study the prevalent assumption that RAE (using pretrained representation as encoder) replaces representation alignment (REPA), which distills the same representation to intermediate layers instead. Through large-scale empirical analysis, we uncover a surprising finding: RAE and REPA exhibit complementary working mechanisms, allowing the same representation to be used as both encoder and target for intermediate diffusion layers. Finally, the original RAE struggles with classifier-free guidance (CFG) and requires training a second, weaker diffusion model for AutoGuidance (AG). We show that REPA itself can be viewed as x-prediction in RAE latent space. By simply re-parameterizing the output of the DiT model, it can provide guidance for "free". Overall, RAEv2 leads to more than 10x faster convergence over the original RAE, achieving a state-of-the-art gFID of 1.06 in just 80 epochs on ImageNet-256. On FDr6, RAEv2 achieves a state-of-the-art 2.17 at just 80 epochs compared to the previous best 3.26 (800 epochs) without any post-training. This motivates EPFID@k (epochs to reach unguided gFID < k) as a measure of training efficiency. RAEv2 attains an EPFID@2 of 35 epochs, versus 177 for the original RAE. We also validate our approach across diverse settings for text-to-image generation and navigation world models, showing consistent improvements. The code is available at https://raev2.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。