通过动态调节注意力注入强度，实现无需训练的图像驱动视频编辑。
FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing
- 根据首帧差异生成编辑掩码，动态控制注意力注入强度。
- 在不依赖训练的情况下，显著提升视频编辑质量与一致性。
- 适合需要快速、零样本视频编辑的场景，如影视后期处理。
图像驱动视频编辑旨在将修改后的首帧内容传播至后续所有帧。现有方法通常使用预训练的图像到视频（I2V）模型对源视频进行反演生成噪声，并利用编辑后的首帧引导采样过程。为保持源视频的运动和布局，常在重建过程中通过注入注意力来干预去噪流程。然而，这种注入方式往往导致效果不佳：过度注入会引发与源视频语义冲突，而注入不足则限制了源视频的表征能力。针对此问题，我们提出一种编辑感知的注意力注入方法（REE），可动态调节每个令牌的注入强度。首先，计算源首帧与编辑后首帧的像素差异，生成对应的编辑掩码；随后，通过光流将首帧掩码传播至整个视频；最后，依据该掩码生成各令牌的编辑感知注入强度，避免在编辑区域进行注入。基于此，我们进一步构建了基于最新涌现的修正流模型的零样本图像驱动视频编辑框架——FREE-Edit。无需微调或训练，FREE-Edit在多种图像驱动视频编辑任务中表现优异，生成结果质量优于现有技术。
原文摘要 · Abstract (English)
Image-driven video editing aims to propagate edit contents from the modified first frame to the remaining frames. Existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics with the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate the injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted in editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: https://free-edit.github.io/page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。