用频域分析优化视频扩散模型,实现高效精准的动态编辑。
FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
- 通过频域感知分解,挖掘预训练模型中的视频先验
- 支持无需重训练的高效编辑,保持时空结构一致
- 适合需要快速修改动作或动态的视频创作者
近期扩散框架在视频编辑方面取得显著进展,实现了高保真与文本提示的良好对齐。然而,传统基于图像扩散模型的方法难以处理视频动态,尤其在运动调整等复杂时序编辑任务上表现不足。尽管现有视频扩散模型能生成高质量结果,但其高昂的计算成本限制了此前图像编辑技术的直接应用。为此,我们提出FADE——一种无需训练却高效的视频编辑方法,通过频域感知分解充分挖掘预训练视频扩散模型中的内在先验。我们首先分析模型注意力模式,揭示视频先验在不同组件中的分布规律;在此基础上,提出分解策略以优化各组件的专用功能。进一步设计频谱引导调制,利用频域线索精炼采样轨迹,避免信息泄露,实现高效、灵活的编辑,同时保留基本的空间与时间结构。在真实视频上的大量实验表明,本方法在定性和定量评估中均持续生成高质量、逼真且时序连贯的编辑结果。代码已开源:https://github.com/EternalEvan/FADE。
原文摘要 · Abstract (English)
Recent advancements in diffusion frameworks have significantly enhanced video editing, achieving high fidelity and strong alignment with textual prompts. However, conventional approaches using image diffusion models fall short in handling video dynamics, particularly for challenging temporal edits like motion adjustments. While current video diffusion models produce high-quality results, adapting them for efficient editing remains difficult due to the heavy computational demands that prevent the direct application of previous image editing techniques. To overcome these limitations, we introduce FADE, a training-free yet highly effective video editing approach that fully leverages the inherent priors from pre-trained video diffusion models via frequency-aware factorization. Rather than simply using these models, we first analyze the attention patterns within the video model to reveal how video priors are distributed across different components. Building on these insights, we propose a factorization strategy to optimize each component's specialized role. Furthermore, we devise spectrum-guided modulation to refine the sampling trajectory with frequency domain cues, preventing information leakage and supporting efficient, versatile edits while preserving the basic spatial and temporal structure. Extensive experiments on real-world videos demonstrate that our method consistently delivers high-quality, realistic and temporally coherent editing results both qualitatively and quantitatively. Code is available at https://github.com/EternalEvan/FADE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。