动态缓存加速扩散Transformer,减少计算量且保持生成质量。
ProCache: Constraint-Aware Feature Caching with Selective Computation for Diffusion Transformer Acceleration
- 根据特征变化规律,自动生成非均匀缓存策略。
- 在深度块和重要标记上选择性计算,抑制误差累积。
- 无需训练,适用于图像生成等实时应用。
扩散Transformer(DiTs)在生成建模中达到顶尖性能,但高计算成本阻碍了实时部署。尽管特征缓存通过利用时间冗余提供了无训练加速方案,现有方法存在两大缺陷:(1) 均匀缓存间隔无法匹配DiT的非均匀时间动态;(2) 过大缓存间隔导致的盲目特征复用会引发严重误差累积。本文分析了去噪过程中DiT特征的演变,发现特征变化与误差传播均具有显著的时间与深度依赖性。受此启发,我们提出ProCache,一种无训练的动态特征缓存框架,包含两个核心组件:(i) 约束感知缓存模式搜索模块,通过离线约束采样生成适配模型时间特性的非均匀激活调度;(ii) 选择性计算模块,在深层块和高重要性标记上选择性计算缓存段,以最小开销缓解误差累积。在PixArt-alpha和DiT上的大量实验表明,ProCache实现了最高1.96倍和2.90倍的加速,且质量损失可忽略,显著优于先前基于缓存的方法。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in generative modeling, yet their high computational cost hinders real-time deployment. While feature caching offers a promising training-free acceleration solution by exploiting temporal redundancy, existing methods suffer from two key limitations: (1) uniform caching intervals fail to align with the non-uniform temporal dynamics of DiT, and (2) naive feature reuse with excessively large caching intervals can lead to severe error accumulation. In this work, we analyze the evolution of DiT features during denoising and reveal that both feature changes and error propagation are highly time- and depth-varying. Motivated by this, we propose ProCache, a training-free dynamic feature caching framework that addresses these issues via two core components: (i) a constraint-aware caching pattern search module that generates non-uniform activation schedules through offline constrained sampling, tailored to the model's temporal characteristics; and (ii) a selective computation module that selectively computes within deep blocks and high-importance tokens for cached segments to mitigate error accumulation with minimal overhead. Extensive experiments on PixArt-alpha and DiT demonstrate that ProCache achieves up to 1.96x and 2.90x acceleration with negligible quality degradation, significantly outperforming prior caching-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。