无需训练即可移除视频中物体及关联效果,保持画面连贯真实。
Object-WIPER : Training-Free Object and Associated Effect Removal in Videos
- 利用预训练扩散模型,通过文本提示定位目标物体与视觉效应区域。
- 在多帧视频中实现干净移除与时间连续的修复,优于现有方法。
- 适合需要快速编辑视频且无训练资源的研究者或创作者使用。
本文提出 Object-WIPER,一种无需训练的视频物体及其关联视觉效果移除框架,可对视频中的动态物体及伴随效果进行精准去除,并用语义一致、时间连贯的内容进行修复。该方法基于预训练文本到视频扩散变换器(DiT),结合用户提供的物体掩码和描述目标物体及效果的查询词,通过视觉-文本交叉注意力与视觉自注意力定位相关视觉令牌,生成中间效果掩码,融合用户掩码后得到最终前景令牌掩码用于替换。首先通过 DiT 反向重构视频以获取结构化噪声,再将掩码区域重新初始化为高斯噪声,同时保留背景令牌;在去噪过程中,将反演阶段保存的背景令牌值复制回,确保场景一致性。针对评估缺失问题,引入新指标,奖励帧间前景令牌的时间一致性、帧内前景与背景令牌的语义一致性,以及输入输出前景令牌的差异性。在 DAVIS 和新构建的真实世界关联效果基准(WIPER-Bench)上的实验表明,Object-WIPER 在该指标上超越了所有训练型与非训练型基线,实现无重训练下的干净移除与稳定重建。相关基准、源代码与预训练模型将公开发布。
原文摘要 · Abstract (English)
In this paper, we introduce Object-WIPER, a training-free framework for removing dynamic objects and their associated visual effects from videos, and inpainting them with semantically consistent and temporally coherent content. Our approach leverages a pre-trained text-to-video diffusion transformer (DiT). Given an input video, a user-provided object mask, and query tokens describing the target object and its effects, we localize relevant visual tokens via visual-text cross-attention and visual self-attention. This produces an intermediate effect mask that we fuse with the user mask to obtain a final foreground token mask to replace. We first invert the video through the DiT to obtain structured noise, then reinitialize the masked tokens with Gaussian noise while preserving background tokens. During denoising, we copy values for the background tokens saved during inversion to maintain scene fidelity. To address the lack of suitable evaluation, we introduce a new object removal metric that rewards temporal consistency among foreground tokens across consecutive frames, coherence between foreground and background tokens within each frame, and dissimilarity between the input and output foreground tokens. Experiments on DAVIS and a newly curated real-world associated effect benchmark (WIPER-Bench) show that Object-WIPER surpasses both training-based and training-free baselines in terms of the metric, achieving clean removal and temporally stable reconstruction without any retraining. Our new benchmark, source code, and pre-trained models will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。