arXiv:2412.18911cs.LGcs.AI2024-12被引 19

提出双策略缓存机制,显著加速扩散Transformer生成。

Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Caching

  • 交替使用激进与保守缓存策略,随机选择计算令牌
  • 在多个模型上实现明显加速,优于传统按重要性缓存
  • 挑战了重要令牌必须持续计算的常识,适合高效生成场景

扩散Transformer(DiT)已成为图像和视频生成的主流方法,但仍面临高昂的计算成本。特征缓存通过缓存前一时刻的特征并在后续时刻复用,跳过部分计算以实现加速。其中,基于令牌的特征缓存根据令牌重要性动态调整缓存比例,旨在跳过不重要令牌的计算而保留重要令牌。本文针对两个问题重新审视该方法的有效性:(1) 是否必须在每一步都计算所谓的‘重要’令牌?(2) 所谓‘重要’令牌真的重要吗?出人意料的是,研究发现持续计算选定的‘重要’令牌并非必要,其选择往往无效,甚至性能劣于随机选择。基于此,本文提出双特征缓存机制(DuCa),交替执行激进与保守缓存策略,并随机选择需计算的令牌。大量实验表明,该方法在DiT、PixArt、FLUX和OpenSora中均显著优于现有令牌级缓存方法。

原文摘要 · Abstract (English)

Diffusion Transformers (DiT) have become the dominant methods in image and video generation yet still suffer substantial computational costs. As an effective approach for DiT acceleration, feature caching methods are designed to cache the features of DiT in previous timesteps and reuse them in the next timesteps, allowing us to skip the computation in the next timesteps. Among them, token-wise feature caching has been introduced to perform different caching ratios for different tokens in DiTs, aiming to skip the computation for unimportant tokens while still computing the important ones. In this paper, we propose to carefully check the effectiveness in token-wise feature caching with the following two questions: (1) Is it really necessary to compute the so-called "important" tokens in each step? (2) Are so-called important tokens really important? Surprisingly, this paper gives some counter-intuition answers, demonstrating that consistently computing the selected ``important tokens'' in all steps is not necessary. The selection of the so-called ``important tokens'' is often ineffective, and even sometimes shows inferior performance than random selection. Based on these observations, this paper introduces dual feature caching referred to as DuCa, which performs aggressive caching strategy and conservative caching strategy iteratively and selects the tokens for computing randomly. Extensive experimental results demonstrate the effectiveness of our method in DiT, PixArt, FLUX, and OpenSora, demonstrating significant improvements than the previous token-wise feature caching.

扩散模型加速生成特征缓存Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。