用扩散Transformer快速修复视频缺失区域，效果好且时序一致。
EraserDiT: Fast Video Inpainting with Diffusion Transformer Model
- 结合扩散模型与Transformer，用循环位置偏移提升长期时序一致性。
- 2160×2100分辨率、97帧视频仅需65秒完成修复，无需加速。
- 支持交互式物体删除并生成提示，适合视频编辑与修复场景。
视频对象去除与修复是计算机视觉和多媒体处理中的关键任务，旨在恢复视频序列中缺失或损坏的区域。传统方法主要依赖基于光流的传播和时空Transformer，但这些方法在利用长期时序特征和保证修复结果的时序一致性方面存在局限，尤其在大掩码情况下性能不佳。为此，本文提出一种基于扩散Transformer（DiT）的新视频修复方法。DiT融合扩散模型与Transformer架构的优势，在保持长期时序一致性的同时实现高质量修复。我们提出循环位置偏移策略，在推理阶段进一步增强时序一致性。此外，该方法支持交互式对象移除并生成对应提示。实验表明，该方法在内容保真度、纹理恢复和时序一致性方面均表现优异。测试在单张NVIDIA H800 GPU上，对2160×2100分辨率、97帧的视频仅需65秒完成修复，无任何加速手段。
原文摘要 · Abstract (English)
Video object removal and inpainting are critical tasks in the fields of computer vision and multimedia processing, aimed at restoring missing or corrupted regions in video sequences. Traditional methods predominantly rely on flow-based propagation and spatio-temporal Transformers, but these approaches face limitations in effectively leveraging long-term temporal features and ensuring temporal consistency in the completion results, particularly when dealing with large masks. Consequently, performance on extensive masked areas remains suboptimal. To address these challenges, this paper introduces a novel video inpainting approach leveraging the Diffusion Transformer (DiT). DiT synergistically combines the advantages of diffusion models and transformer architectures to maintain long-term temporal consistency while ensuring high-quality inpainting results. We propose a Circular Position-Shift strategy to further enhance long-term temporal consistency during the inference stage. Additionally, the proposed method interactively removes specified objects, and generates corresponding prompts. In terms of processing speed, it takes only 65 seconds (testing on one NVIDIA H800 GPU) to complete a video with a resolution of $2160 \times 2100$ with 97 frames without any acceleration method. Experimental results indicate that the proposed method demonstrates superior performance in content fidelity, texture restoration, and temporal consistency. Project page:https://jieliu95.github.io/EraserDiT_demo/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。