通过分离主成分与残差，实现扩散模型高效特征缓存
Forecast the Principal, Stabilize the Residual: Subspace-Aware Feature Caching for Efficient Diffusion Transformers
- 用SVD分解特征，区分平滑演化的主成分和波动的残差
- 主成分用EMA预测，残差直接复用，实现5.55倍加速
- 兼容蒸馏、量化等技术，适合部署加速场景
扩散变换器（DiT）在图像与视频生成中达到前所未有的质量，但其迭代采样过程计算开销巨大。为加速推理，特征缓存方法通过复用跨时间步的中间表示应运而生。然而，现有方法对所有特征成分一视同仁。我们发现DiT特征空间包含性质迥异的主成分与残差子空间：主成分演化平稳可预测，残差则呈现高波动性、低能量的振荡，难以准确预测。基于此洞察，我们提出SVD-Cache——一种子空间感知的缓存框架，通过奇异值分解（SVD）分解扩散特征，对主导的低秩成分使用指数移动平均（EMA）预测，并直接复用残差子空间。大量实验表明，SVD-Cache在多种模型与方法上实现近乎无损加速，如在FLUX和HunyuanVideo上分别实现5.55×加速，且兼容模型加速技术如知识蒸馏、量化与稀疏注意力。代码已附于补充材料，将发布于Github。
原文摘要 · Abstract (English)
Diffusion Transformer (DiT) models have achieved unprecedented quality in image and video generation, yet their iterative sampling process remains computationally prohibitive. To accelerate inference, feature caching methods have emerged by reusing intermediate representations across timesteps. However, existing caching approaches treat all feature components uniformly. We reveal that DiT feature spaces contain distinct principal and residual subspaces with divergent temporal behavior: the principal subspace evolves smoothly and predictably, while the residual subspace exhibits volatile, low-energy oscillations that resist accurate prediction. Building on this insight, we propose SVD-Cache, a subspace-aware caching framework that decomposes diffusion features via Singular Value Decomposition (SVD), applies exponential moving average (EMA) prediction to the dominant low-rank components, and directly reuses the residual subspace. Extensive experiments demonstrate that SVD-Cache achieves near-lossless across diverse models and methods, including 5.55$\times$ speedup on FLUX and HunyuanVideo, and compatibility with model acceleration techniques including distillation, quantization and sparse attention. Our code is in supplementary material and will be released on Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。