arXiv:2504.14335cs.CVcs.AI2025-04CVPR被引 11

无需反演,用视觉提示实现单帧编辑的视频一致性生成

Visual Prompting for One-shot Controllable Video Editing without Inversion

  • 用视觉提示替代传统反演,直接在图像空间进行编辑传播
  • 提出内容一致采样,确保生成帧与源帧内容一致
  • 适合需要快速、高质量视频编辑的创作者和研究人员

单帧可控视频编辑(OCVE)是一项重要但具有挑战性的任务,旨在将用户在视频首帧上使用任意图像编辑工具完成的修改,传播到后续所有帧,同时保证编辑帧与源帧的内容一致性。现有方法依赖DDIM反演将源帧转换为潜在噪声,再通过预训练扩散模型,以用户编辑的首帧为条件生成编辑视频。然而,DDIM反演过程会累积误差,导致潜在噪声无法准确重建源帧,最终损害生成帧的内容一致性。为此,本文提出一种新视角:通过视觉提示实现无反演的OCVE。受一致性模型启发,我们设计了内容一致采样(CCS),确保生成帧与源帧内容一致;进一步提出基于Stein变分梯度下降的时序-内容一致采样(TCS),保障编辑帧间的时序一致性。大量实验验证了该方法的有效性。

原文摘要 · Abstract (English)

One-shot controllable video editing (OCVE) is an important yet challenging task, aiming to propagate user edits that are made -- using any image editing tool -- on the first frame of a video to all subsequent frames, while ensuring content consistency between edited frames and source frames. To achieve this, prior methods employ DDIM inversion to transform source frames into latent noise, which is then fed into a pre-trained diffusion model, conditioned on the user-edited first frame, to generate the edited video. However, the DDIM inversion process accumulates errors, which hinder the latent noise from accurately reconstructing the source frames, ultimately compromising content consistency in the generated edited frames. To overcome it, our method eliminates the need for DDIM inversion by performing OCVE through a novel perspective based on visual prompting. Furthermore, inspired by consistency models that can perform multi-step consistency sampling to generate a sequence of content-consistent images, we propose a content consistency sampling (CCS) to ensure content consistency between the generated edited frames and the source frames. Moreover, we introduce a temporal-content consistency sampling (TCS) based on Stein Variational Gradient Descent to ensure temporal consistency across the edited frames. Extensive experiments validate the effectiveness of our approach.

视频编辑扩散模型视觉提示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。