用预训练VAE特征引导扩散模型,训练更快更高效。
SRA 2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training
- 用轻量投影层对齐扩散模型与VAE的中间特征
- 训练速度更快,生成质量提升,仅增4%计算量
- 无需外部编码器或双模型,适合追求高效训练的研究者
基于去噪的扩散变换器虽生成效果优异,但训练收敛效率低。现有方法如REPA依赖外部表示编码器,SRA需双模型结构,均带来额外计算开销。本文提出SRA 2,一种轻量级内在引导框架,利用现成预训练变分自编码器(VAE)特征:其重建特性能天然编码纹理细节、结构模式和基础语义信息。SRA 2通过轻量投影层将扩散变换器的中间潜在特征与VAE特征对齐,以特征对齐损失监督。该设计无需外部编码器或双模型维护,实现简单高效的训练流程。大量实验表明,SRA 2在生成质量与训练收敛速度上均优于原始扩散模型,达到或超过现有加速方法水平,仅增加4%额外GFLOPs,且无外部引导模型的额外成本。
原文摘要 · Abstract (English)
Denoising-based diffusion transformers, despite their strong generation performance, suffer from inefficient training convergence. Existing methods addressing this issue, such as REPA (relying on external representation encoders) or SRA (requiring dual-model setups), inevitably incur heavy computational overhead during training due to external dependencies. To tackle these challenges, this paper proposes SRA 2, a lightweight intrinsic guidance framework for efficient diffusion training. SRA 2 leverages off-the-shelf pre-trained Variational Autoencoder (VAE) features: their reconstruction property ensures inherent encoding of visual priors like rich texture details, structural patterns, and basic semantic information. Specifically, SRA 2 aligns the intermediate latent features of diffusion transformers with VAE features via a lightweight projection layer, supervised by a feature alignment loss. This design accelerates training without extra representation encoders or dual-model maintenance, resulting in a simple yet effective pipeline. Extensive experiments demonstrate that SRA 2 improves both generation quality and training convergence speed compared to vanilla diffusion transformers, matches or outperforms state-of-the-art acceleration methods, and incurs merely 4% extra GFLOPs with zero additional cost for external guidance models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。