arXiv:2507.01467cs.CV2025-07NeurIPS被引 86

用单一类别标记纠缠低层图像隐变量,让扩散Transformer训练更快更稳。

Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think

  • 将预训练模型的类别标记与图像隐变量纠缠,指导去噪过程。
  • 在ImageNet上训练速度提升63倍,40万步效果超传统方法400万步。
  • 仅增加不到0.5%计算量,适合高效生成任务和资源受限场景。

REPA及其变体通过引入预训练模型的外部视觉表示,缓解扩散模型训练难题,其核心是使去噪网络的噪声隐藏投影与基础清晰图像表示对齐。然而,这种外部对齐在去噪推理过程中并不存在,未能充分挖掘判别性表示的潜力。本文提出一种简单有效的方法——生成用表示纠缠(REG),将低层图像隐变量与预训练模型的单个高层类别标记进行纠缠以实现去噪。REG可直接从纯噪声生成一致的图像-类别对,显著提升生成质量与训练效率。该方法仅需额外一个标记,推理开销极小(FLOPs与延迟增加不足0.5%)。推理过程同时重建图像隐变量与全局语义,获得的语义知识主动引导图像生成。在ImageNet 256×256上,SiT-XL/2 + REG相比SiT-XL/2与SiT-XL/2 + REPA分别实现63倍与23倍加速;更令人惊叹的是,仅训练40万步的SiT-L/2 + REG超越了训练400万步的SiT-XL/2 + REPA(10倍时长)。代码已开源:https://github.com/Martinser/REG。

原文摘要 · Abstract (English)

REPA and its variants effectively mitigate training challenges in diffusion models by incorporating external visual representations from pretrained models, through alignment between the noisy hidden projections of denoising networks and foundational clean image representations. We argue that the external alignment, which is absent during the entire denoising inference process, falls short of fully harnessing the potential of discriminative representations. In this work, we propose a straightforward method called Representation Entanglement for Generation (REG), which entangles low-level image latents with a single high-level class token from pretrained foundation models for denoising. REG acquires the capability to produce coherent image-class pairs directly from pure noise, substantially improving both generation quality and training efficiency. This is accomplished with negligible additional inference overhead, requiring only one single additional token for denoising (<0.5\% increase in FLOPs and latency). The inference process concurrently reconstructs both image latents and their corresponding global semantics, where the acquired semantic knowledge actively guides and enhances the image generation process. On ImageNet 256$\times$256, SiT-XL/2 + REG demonstrates remarkable convergence acceleration, achieving $\textbf{63}\times$ and $\textbf{23}\times$ faster training than SiT-XL/2 and SiT-XL/2 + REPA, respectively. More impressively, SiT-L/2 + REG trained for merely 400K iterations outperforms SiT-XL/2 + REPA trained for 4M iterations ($\textbf{10}\times$ longer). Code is available at: https://github.com/Martinser/REG.

扩散模型生成模型训练加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。