统一视频补全与物体插入,支持长视频多模态控制。
MTV-Inpaint: Multi-Task Long Video Inpainting
- 设计双分支空间注意力,融合补全与插入任务
- 支持文本、图像等多模态输入,可处理数百帧长视频
- 适用于物体编辑、移除、长视频修复等多样场景
视频修复需在视频中修改局部区域并保持时空一致性。现有方法多聚焦于场景补全,难以可控地插入新物体。尽管文本到视频扩散模型为文本引导修复提供了可能,但直接应用仍存在任务不统一、控制能力弱、难处理长视频等问题。为此,我们提出MTV-Inpaint,一个统一的多任务视频修复框架,可同时实现传统场景补全与新颖物体插入。通过在T2V扩散U-Net中设计双分支空间注意力机制,实现两类任务的无缝集成。除文本引导外,还引入图像到视频(I2V)修复模式,融合多种图像修复模型以支持多模态控制。进一步提出两阶段流程:关键帧修复 + 逐帧传播,有效处理数百帧长视频。大量实验表明,MTV-Inpaint在补全与插入任务上均达到当前最优性能,并展现出在多模态修复、物体编辑、移除、图像物体绘制等衍生应用中的强大泛化能力。项目页:https://mtv-inpaint.github.io/。
原文摘要 · Abstract (English)
Video inpainting involves modifying local regions within a video, ensuring spatial and temporal consistency. Most existing methods focus primarily on scene completion (i.e., filling missing regions) and lack the capability to insert new objects into a scene in a controllable manner. Fortunately, recent advancements in text-to-video (T2V) diffusion models pave the way for text-guided video inpainting. However, directly adapting T2V models for inpainting remains limited in unifying completion and insertion tasks, lacks input controllability, and struggles with long videos, thereby restricting their applicability and flexibility. To address these challenges, we propose MTV-Inpaint, a unified multi-task video inpainting framework capable of handling both traditional scene completion and novel object insertion tasks. To unify these distinct tasks, we design a dual-branch spatial attention mechanism in the T2V diffusion U-Net, enabling seamless integration of scene completion and object insertion within a single framework. In addition to textual guidance, MTV-Inpaint supports multimodal control by integrating various image inpainting models through our proposed image-to-video (I2V) inpainting mode. Additionally, we propose a two-stage pipeline that combines keyframe inpainting with in-between frame propagation, enabling MTV-Inpaint to effectively handle long videos with hundreds of frames. Extensive experiments demonstrate that MTV-Inpaint achieves state-of-the-art performance in both scene completion and object insertion tasks. Furthermore, it demonstrates versatility in derived applications such as multi-modal inpainting, object editing, removal, image object brush, and the ability to handle long videos. Project page: https://mtv-inpaint.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。