arXiv:2504.03140cs.CV2025-04ICCV被引 15

通过分析注意力分布,动态区分前景与背景,实现视频扩散模型的智能缓存。

Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Models

  • 根据注意力分布识别前景/背景区块,分层设计缓存策略。
  • 在Wan2.1上实现2.01倍加速,视觉质量无损失。
  • 适合需要高效视频生成的应用场景,如实时渲染。

近期扩散模型在视频生成方面展现出强大能力,但计算开销仍是实际应用的主要挑战。尽管特征缓存被提出以降低扩散模型的计算负担,但现有方法通常忽略各模块的重要性差异,导致重用效率低下且输出质量下降。为此,我们提出ProfilingDiT,一种新型自适应缓存策略,显式分离关注前景与背景的模块。通过对扩散模型中注意力分布的系统分析,我们发现:1)多数层对前景或背景区域具有稳定偏好;2)初始阶段预测噪声的跨步相似性较低,随着去噪过程推进逐渐稳定。基于此,我们设计了一种选择性缓存策略:对动态前景保留完整计算,对静态背景特征进行高效缓存。该方法显著降低计算开销的同时保持视觉保真度。大量实验表明,本框架在多项质量指标下保持高保真,实现显著加速(如在Wan2.1上达2.01倍),为高效视频生成提供了可行方案。

原文摘要 · Abstract (English)

Recent advances in diffusion models have demonstrated remarkable capabilities in video generation. However, the computational intensity remains a significant challenge for practical applications. While feature caching has been proposed to reduce the computational burden of diffusion models, existing methods typically overlook the heterogeneous significance of individual blocks, resulting in suboptimal reuse and degraded output quality. To this end, we address this gap by introducing ProfilingDiT, a novel adaptive caching strategy that explicitly disentangles foreground and background-focused blocks. Through a systematic analysis of attention distributions in diffusion models, we reveal a key observation: 1) Most layers exhibit a consistent preference for either foreground or background regions. 2) Predicted noise shows low inter-step similarity initially, which stabilizes as denoising progresses. This finding inspires us to formulate a selective caching strategy that preserves full computation for dynamic foreground elements while efficiently caching static background features. Our approach substantially reduces computational overhead while preserving visual fidelity. Extensive experiments demonstrate that our framework achieves significant acceleration (e.g., 2.01 times speedup for Wan2.1) while maintaining visual fidelity across comprehensive quality metrics, establishing a viable method for efficient video generation.

视频生成扩散模型缓存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。