arXiv:2503.18033cs.CV2025-03SIGGRAPH被引 12

无需训练,用预训练视频扩散模型快速实现视频对象分离与重合成。

OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models

论文配图:OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models
图 1 · 摘自论文原文
  • 基于预训练扩散模型,通过时空调节机制实现零样本视频去物。
  • 可实时处理,背景重建质量高且保持时间一致性。
  • 适合影视后期、视频编辑等需要快速生成新视频的场景。

Omnimatte 的目标是将视频分解为语义明确的层,包括背景和各个物体及其附带效果(如阴影、反射)。现有方法通常需要大量训练或昂贵的自监督优化。本文提出 OmnimatteZero,一种无需训练的方法,利用现成的预训练视频扩散模型完成 omnimatte 任务。该方法能移除视频中的物体,提取包含其效果的独立物体层,并将其合成为新视频。为此,我们改进了零样本图像修复技术以应对视频去物任务,引入时空注意力引导模块,确保物体去除准确且背景重建时序一致。进一步发现,自注意力图能捕捉物体及其痕迹信息,用于修复物体效应,留下干净背景。此外,通过简单的潜在空间算术操作,可隔离并无缝重组物体层与新视频层,生成新视频。实验表明,OmnimatteZero 在背景重建质量上表现优异,并创下最快 omni-matte 方法纪录,实现实时性能,单帧运行时间极低。

原文摘要 · Abstract (English)

In Omnimatte, one aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. These are accomplished by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. To overcome this, we introduce temporal and spatial attention guidance modules that steer the diffusion process for accurate object removal and temporally consistent background reconstruction. We further show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.

视频生成扩散模型去物实时处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。