用扩散模型提升视频修复细节与连贯性，解决大区域遮挡时模糊问题。
DiffuEraser: A Diffusion Model for Video Inpainting
- 基于稳定扩散模型，结合先验信息初始化，减少噪声和幻觉。
- 在长序列推理中提升时间一致性，优于现有方法。
- 适合需要高质量视频修复的场景，如影视后期、数据清洗。
近期视频修复算法结合基于流的像素传播与视觉变压器，利用光流从邻近帧恢复纹理和物体，通过视觉变压器填补遮挡区域。然而，面对大范围遮挡时，这些方法常出现模糊和时间不一致问题，凸显对更强生成能力的需求。扩散模型因在图像与视频生成中表现优异而受到关注。本文提出DiffuEraser，一种基于稳定扩散的视频修复模型，旨在更精细地填充遮挡区域并保持结构连贯性。通过引入先验信息进行初始化和弱条件控制，有效缓解噪声伪影并抑制幻觉。为提升长序列推理的时间一致性，我们扩展了先验模型和DiffuEraser的时序感受野，并利用视频扩散模型的时间平滑特性进一步增强一致性。实验表明，所提方法在内容完整性和时间一致性方面均优于当前最优技术，同时保持可接受的效率。
原文摘要 · Abstract (English)
Recent video inpainting algorithms integrate flow-based pixel propagation with transformer-based generation to leverage optical flow for restoring textures and objects using information from neighboring frames, while completing masked regions through visual Transformers. However, these approaches often encounter blurring and temporal inconsistencies when dealing with large masks, highlighting the need for models with enhanced generative capabilities. Recently, diffusion models have emerged as a prominent technique in image and video generation due to their impressive performance. In this paper, we introduce DiffuEraser, a video inpainting model based on stable diffusion, designed to fill masked regions with greater details and more coherent structures. We incorporate prior information to provide initialization and weak conditioning,which helps mitigate noisy artifacts and suppress hallucinations. Additionally, to improve temporal consistency during long-sequence inference, we expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models. Experimental results demonstrate that our proposed method outperforms state-of-the-art techniques in both content completeness and temporal consistency while maintaining acceptable efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。