按令牌敏感度动态缓存特征，加速扩散Transformer生成
Accelerating Diffusion Transformers with Token-wise Feature Caching
- 按令牌对缓存的敏感度差异，智能选择可缓存内容
- 在OpenSora和PixArt-α上实现2.36倍和1.93倍加速
- 无需训练即可适配不同模型结构，适合高效生成场景
扩散变压器在图像和视频生成中表现出显著效果，但计算开销巨大。为解决此问题，特征缓存方法通过缓存前一时刻特征并复用于后续时刻以加速推理。然而，现有方法忽略不同令牌对缓存的敏感度差异，部分令牌缓存可能导致整体生成质量下降10倍以上。本文提出基于令牌的特征缓存机制，可自适应选择最适宜缓存的令牌，并针对不同层级和深度的神经层设置差异化缓存比例。在PixArt-α、OpenSora和DiT上的大量实验表明，该方法在不需训练的情况下，在图像与视频生成中均有效。例如，OpenSora和PixArt-α分别实现2.36倍和1.93倍加速，且生成质量几乎无损。
原文摘要 · Abstract (English)
Diffusion transformers have shown significant effectiveness in both image and video synthesis at the expense of huge computation costs. To address this problem, feature caching methods have been introduced to accelerate diffusion transformers by caching the features in previous timesteps and reusing them in the following timesteps. However, previous caching methods ignore that different tokens exhibit different sensitivities to feature caching, and feature caching on some tokens may lead to 10$\times$ more destruction to the overall generation quality compared with other tokens. In this paper, we introduce token-wise feature caching, allowing us to adaptively select the most suitable tokens for caching, and further enable us to apply different caching ratios to neural layers in different types and depths. Extensive experiments on PixArt-$α$, OpenSora, and DiT demonstrate our effectiveness in both image and video generation with no requirements for training. For instance, 2.36$\times$ and 1.93$\times$ acceleration are achieved on OpenSora and PixArt-$α$ with almost no drop in generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。