提出无需训练的缓存加速方法，提升扩散模型生成效率。
Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition
- 通过增量校准缓存，利用预训练模型自动生成校正参数。
- 在35步DDIM基础上减少45%计算量，提升12分图像质量分数。
- 适合需要高效生成且对精度要求高的图像生成场景。
扩散变换器（DiT）模型凭借出色的生成能力与可扩展性，在图像生成领域取得显著成功。然而，扩散模型的迭代特性导致计算复杂度高，制约了实际部署。现有基于缓存的加速方法虽利用时间相似性跳过冗余计算，但缺乏校正可能导致质量下降。本文提出增量校准缓存（increment-calibrated caching），一种无需训练的DiT加速方法，其校准参数由预训练模型自身通过低秩近似生成。为应对异常激活带来的校正失败问题，引入通道感知奇异值分解（channel-aware SVD），进一步增强校准效果。实验表明，该方法在相近计算资源下始终优于朴素缓存方法。相较于35步DDIM，本方法消除超过45%计算量，图像质量分数（IS）提升12，仅带来小于0.06的FID增加。代码已开源。
原文摘要 · Abstract (English)
Diffusion transformer (DiT) models have achieved remarkable success in image generation, thanks for their exceptional generative capabilities and scalability. Nonetheless, the iterative nature of diffusion models (DMs) results in high computation complexity, posing challenges for deployment. Although existing cache-based acceleration methods try to utilize the inherent temporal similarity to skip redundant computations of DiT, the lack of correction may induce potential quality degradation. In this paper, we propose increment-calibrated caching, a training-free method for DiT acceleration, where the calibration parameters are generated from the pre-trained model itself with low-rank approximation. To deal with the possible correction failure arising from outlier activations, we introduce channel-aware Singular Value Decomposition (SVD), which further strengthens the calibration effect. Experimental results show that our method always achieve better performance than existing naive caching methods with a similar computation resource budget. When compared with 35-step DDIM, our method eliminates more than 45% computation and improves IS by 12 at the cost of less than 0.06 FID increase. Code is available at https://github.com/ccccczzy/icc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。