arXiv:2411.17616cs.CV2024-11ICCV被引 13

通过谱约束长跳跃连接,提升扩散Transformer的稳定性和生成效率

Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints

  • 引入谱约束长跳跃连接,稳定深层特征传播
  • 训练提速4.4倍,推理加速1.5-2倍且质量几乎无损
  • 适合追求高效高质图像视频生成的开发者与研究者

扩散Transformer(DiT)在图像与视频生成中表现卓越,但实际应用受动态特征不稳定的困扰,导致缓存推理时误差累积。通过系统分析,我们发现缺乏长程特征保持机制是特征传播不稳定和扰动敏感的根本原因。为此,提出Skip-DiT,一种增强长跳跃连接(LSCs)的DiT变体,其核心为类U-Net的长跳跃结构。理论谱范数分析与可视化表明,LSCs能有效稳定特征动态。该架构支持高效的静态缓存机制,可在跨时间步复用深层特征,仅更新浅层组件。在图像与视频生成任务上,实验显示Skip-DiT实现:(1) 训练加速4.4倍且收敛更快;(2) 推理加速1.5-2倍,质量损失极小,优于现有DiT缓存方法。结果确立长跳跃连接对稳定高效扩散Transformer的关键作用。代码已开源。

原文摘要 · Abstract (English)

Diffusion Transformers (DiT) have emerged as a powerful architecture for image and video generation, offering superior quality and scalability. However, their practical application suffers from inherent dynamic feature instability, leading to error amplification during cached inference. Through systematic analysis, we identify the absence of long-range feature preservation mechanisms as the root cause of unstable feature propagation and perturbation sensitivity. To this end, we propose Skip-DiT, an image and video generative DiT variant enhanced with Long-Skip-Connections (LSCs) - the key efficiency component in U-Nets. Theoretical spectral norm and visualization analysis demonstrate how LSCs stabilize feature dynamics. Skip-DiT architecture and its stabilized dynamic feature enable an efficient statical caching mechanism that reuses deep features across timesteps while updating shallow components. Extensive experiments across the image and video generation tasks demonstrate that Skip-DiT achieves: (1) 4.4 times training acceleration and faster convergence, (2) 1.5-2 times inference acceleration with negligible quality loss and high fidelity to the original output, outperforming existing DiT caching methods across various quantitative metrics. Our findings establish Long-Skip-Connections as critical architectural components for stable and efficient diffusion transformers. Codes are provided in the https://github.com/OpenSparseLLMs/Skip-DiT.

扩散模型Transformer生成效率特征稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。