通过动态感知的令牌剪枝，让Stable Diffusion生成提速9倍且画质更好。
Token Pruning for Caching Better: 9 Times Acceleration on Stable Diffusion for Free
- 只保留高动态令牌参与注意力计算，提升特征时序变化性。
- ImageNet上加速9倍，FID降低0.33；COCO-30k上加速7倍，FID降2.17。
- 无需训练，可直接用于现有Stable Diffusion模型，适合加速生成场景。
Stable Diffusion在文本到图像生成领域取得显著成功，其强大的生成能力与多样结果产生深远影响。然而，迭代去噪过程带来高昂计算开销，拖慢生成速度，限制了广泛应用。社区已提出多种降本方法，其中特征缓存因高效简洁备受关注。但直接复用前步特征会导致相邻时间步特征趋同，削弱特征时序动态性，最终损害生成质量。本文提出动态感知令牌剪枝（DaTo）方法，通过筛选低动态令牌，仅让高动态令牌参与自注意力层，延长特征跨时间步的动态范围。该方法以无训练方式结合特征缓存与令牌剪枝，实现时空双重信息复用。应用于ImageNet上的Stable Diffusion，实现9倍加速，同时FID降低0.33，表明画质提升；在COCO-30k上实现7倍加速，FID下降2.17。
原文摘要 · Abstract (English)
Stable Diffusion has achieved remarkable success in the field of text-to-image generation, with its powerful generative capabilities and diverse generation results making a lasting impact. However, its iterative denoising introduces high computational costs and slows generation speed, limiting broader adoption. The community has made numerous efforts to reduce this computational burden, with methods like feature caching attracting attention due to their effectiveness and simplicity. Nonetheless, simply reusing features computed at previous timesteps causes the features across adjacent timesteps to become similar, reducing the dynamics of features over time and ultimately compromising the quality of generated images. In this paper, we introduce a dynamics-aware token pruning (DaTo) approach that addresses the limitations of feature caching. DaTo selectively prunes tokens with lower dynamics, allowing only high-dynamic tokens to participate in self-attention layers, thereby extending feature dynamics across timesteps. DaTo combines feature caching with token pruning in a training-free manner, achieving both temporal and token-wise information reuse. Applied to Stable Diffusion on the ImageNet, our approach delivered a 9$\times$ speedup while reducing FID by 0.33, indicating enhanced image quality. On the COCO-30k, we observed a 7$\times$ acceleration coupled with a notable FID reduction of 2.17.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。