用高效DiT模型实现高质量视频修复，支持任意长度视频。
DiTPainter: Efficient Video Inpainting with Diffusion Transformers
- 基于自研轻量DiT架构，从零训练不依赖大模型。
- 在任意长度视频上修复效果优于现有方法，时空一致性强。
- 适合需要快速生成高质量修复视频的场景。
现有视频修复算法多依赖光流构建映射关系，通过相邻帧像素传播填补缺失区域。尽管该机制有效，但在光流不准确或遮挡区域较大时易出现模糊和不一致问题。近期，扩散变换器（Diffusion Transformer, DiT）在视频生成任务中表现出色，但预训练的DiT模型参数量庞大，难以高效应用于视频修复。本文提出DiTPainter，一种基于扩散变换器的端到端视频修复模型。该模型采用专为视频修复设计的高效变换器网络，从零开始训练，无需依赖任何大型预训练模型。DiTPainter可处理任意长度视频，适用于视频去字幕与视频补全任务，计算开销合理。实验表明，其在修复质量与时空一致性方面均超越现有算法。
原文摘要 · Abstract (English)
Many existing video inpainting algorithms utilize optical flows to construct the corresponding maps and then propagate pixels from adjacent frames to missing areas by mapping. Despite the effectiveness of the propagation mechanism, they might encounter blurry and inconsistencies when dealing with inaccurate optical flows or large masks. Recently, Diffusion Transformer (DiT) has emerged as a revolutionary technique for video generation tasks. However, pretrained DiT models for video generation all contain a large amount of parameters, which makes it very time consuming to apply to video inpainting tasks. In this paper, we present DiTPainter, an end-to-end video inpainting model based on Diffusion Transformer (DiT). DiTPainter uses an efficient transformer network designed for video inpainting, which is trained from scratch instead of initializing from any large pretrained models. DiTPainter can address videos with arbitrary lengths and can be applied to video decaptioning and video completion tasks with an acceptable time cost. Experiments show that DiTPainter outperforms existing video inpainting algorithms with higher quality and better spatial-temporal consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。