通过多维缓存优化扩散模型推理，显著提速且不损失画质。
OmniCache: Multidimensional Hierarchical Feature Caching For Diffusion Models
- 基于四类冗余设计分层缓存机制，复用中间特征减少计算。
- 在SD3、SVD-XT、Latte上分别提速35%、25%、28%，保持画质与运动连贯性。
- 无需训练，适用于图像和视频生成，适合高分辨率生成场景。
高分辨率图像与视频扩散模型（如SD3、FLUX及近期视频扩散变压器）虽生成质量大幅提升，但推理成本高昂，因需在多个采样步骤中反复计算注意力密集的去噪器。本文通过利用中间扩散特征中的冗余，而非修改模型权重或重新训练，解决此问题。我们识别出四类互补冗余：帧内、帧间、运动与去噪步冗余。基于此，提出统一的分层缓存框架OmniCache，通过Token Cache、Frame Cache、Block Cache与Layered Cache实现多维特征重用。不同于仅平均匹配特征的基线方法，OmniCache采用相似性匹配选择可缓存特征，跳过冗余计算，并恢复位置一致的缓存激活，保持特征顺序与时空结构。该框架在时空层间复用空间与时间特征，且Layered Cache捕捉模型层级的跨步冗余。在SD3、SVD-XT与Latte上，OmniCache分别将推理延迟降低35%、25%、28%，同时维持视觉保真度与运动连贯性，且为无训练设置。
原文摘要 · Abstract (English)
High-resolution image and video diffusion models, including SD3, FLUX, and recent video diffusion transformers, have substantially improved generative quality but remain expensive at inference time because they repeatedly evaluate attention-heavy denoisers over many sampling steps. We address this inefficiency by exploiting redundancy in intermediate diffusion features rather than changing model weights or retraining. We identify four complementary redundancy sources in image and video generation: intra-frame, inter-frame, motion, and denoising-step redundancy. Based on this analysis, we propose OmniCache, a unified hierarchical caching framework that performs multidimensional feature reuse through Token Cache, Frame Cache, Block Cache, and Layered Cache. Unlike token-merging baselines that average matched features, OmniCache uses similarity matching to select cacheable features, skips redundant computation, and restores positionally consistent cached activations, preserving feature order and spatial-temporal structure. The resulting framework reuses spatial features in temporal layers and temporal features in spatial layers, while Layered Cache captures cross-step redundancy at the model-layer level. Across SD3, SVD-XT, and Latte, OmniCache reduces inference latency by up to 35%, 25%, and 28%, respectively, while maintaining visual fidelity and motion coherence in a training-free setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。