仅用模型内部特征即可加速扩散变压器训练,效果超越依赖外部特征的方法。
Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?
- 用自蒸馏方式让模型内部特征自我优化,替代外部语义引导。
- 在40轮内对齐VAE潜空间特征,再通过无分类器引导提升语义区分度。
- 无需外部依赖,训练更快、生成质量更高,适合追求高效训练的开发者。
近期工作如REPA表明,使用外部语义特征(如DINO)可显著加速扩散变压器(DiT)的训练。然而,依赖预训练外部特征会引入额外依赖。本文认为,DiT本身具备自我引导能力,提出SelfTranscendence方法,仅通过内部特征监督实现快速收敛。理想的内部引导特征需结构清晰以帮助浅层分离噪声与信号,且语义判别性强以支持有效表征学习。为此,我们先在短周期(如40轮)内将DiT特征与原生的潜在扩散中的清洁VAE潜特征对齐,提升其结构表示;随后对中间特征应用无分类器引导,增强其判别能力与语义表达力。这些完全由模型内学习的丰富内部特征作为监督信号,用于从头训练新DiT。相比现有自包含方法,本方法性能显著提升,甚至在类别到图像和文本到图像生成任务中,超越使用外部DINO特征的REPA,在生成质量与收敛速度上均表现更优。
原文摘要 · Abstract (English)
Recent works such as REPA have shown that guiding diffusion models with external semantic features (e.g., DINO) can significantly accelerate the training of diffusion transformers (DiTs). However, the use of pretrained external features as guidance signals introduces additional dependencies. We argue that DiTs actually have the power to guide the training of themselves, and propose SelfTranscendence, an effective method that achieves fast convergence using internal feature supervision only. The desired internal guidance features should meet two requirements: structurally clean to help shallow blocks separate noise from signal, and semantically discriminative to help shallow layers learn effective representations. With this consideration, we first align the DiT features with the clean VAE latent features, a native component of latent diffusion, for a short training phase (e.g., 40 epochs) to improve their structural representations, then apply the classifier-free guidance to the intermediate features, enhancing their discriminative capability and semantic expressiveness. These enriched internal features, learned entirely within the model, are used as supervision signals to guide a new DiT training from scratch. Compared to existing self-contained methods, our approach achieves a significant performance boost. It can even surpass REPA, which uses the external DINO features as guidance, in both generation quality and convergence speed for both class-to-image and text-to-image generation tasks. The source code of our method can be found at https://github.com/csslc/Self-Transcendence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。