arXiv:2601.16208cs.CV2026-01被引 43

用自编码器提升文本到图像生成,效果比传统方法更好更稳定。

Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders

  • 在高维语义空间中训练自编码器,实现大规模文本图像生成。
  • 模型在9.8B参数下收敛更快,且256轮微调仍不崩溃,优于VAE。
  • 共享表示空间让模型能直接推理生成结果,适合统一多模态模型。

表示自编码器(RAEs)在ImageNet上的扩散建模中表现出显著优势,因其在高维语义潜在空间中进行训练。本文探究该框架是否可扩展至大规模、自由形式的文本到图像(T2I)生成。我们首先在冻结的表示编码器(SigLIP-2)上,通过网络、合成数据和文本渲染数据对RAE解码器进行扩展,发现虽然规模提升增强了通用保真度,但特定领域(如文字生成)需针对性数据组合。随后,我们严格测试了原用于ImageNet的RAE设计选择。分析表明,随着规模扩大,框架简化:尽管依赖维度的噪声调度依然关键,但宽扩散头和噪声增强解码等架构复杂性在大尺度下收益极低。基于此简化框架,我们在0.5B至9.8B参数的扩散变压器上,对RAE与当前最先进的FLUX VAE进行了受控对比。结果显示,所有规模下RAE在预训练中持续优于VAE;在高质量数据集微调时,基于VAE的模型在64轮后严重过拟合,而RAE模型在256轮内保持稳定并始终表现更优。整体实验表明,基于RAE的扩散模型收敛更快、生成质量更高,是大规模T2I生成更简单且更强的基础。此外,由于视觉理解与生成可在共享表示空间中运行,多模态模型可直接对生成潜在变量进行推理,为统一模型开辟新可能。

原文摘要 · Abstract (English)

Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.

文本生成扩散模型自编码器多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。