用图片参考编辑视频,保持动作连贯性
Edit as You See: Image-guided Video Editing via Masked Motion Modeling
- 基于图像编辑模型,引入可学习运动模块维持时间一致性
- 通过掩码运动建模提升帧间动态捕捉能力,生成流畅视频
- 适合需要精准视觉控制的视频编辑场景
扩散模型的进展推动了文本引导的视频编辑,但图像引导的视频编辑研究仍较匮乏。本文提出一种新型图像引导视频编辑扩散模型IVEDiff,用户仅需在初始帧标注目标物体并提供参考图片即可编辑视频,无需文本提示。IVEDiff基于图像编辑模型,引入可学习运动模块以保持时序一致性。受自监督学习启发,采用掩码运动建模微调策略,使运动模块能捕捉帧间运动动态,同时保留基础模型对帧内语义关联的建模能力。此外,提出光流引导的运动参考网络,确保帧间信息准确传播,缓解无效信息干扰。我们还构建了一个基准数据集。大量实验表明,该方法能生成时序平滑、高质量的编辑视频,对多种编辑对象均具鲁棒性。
原文摘要 · Abstract (English)
Recent advancements in diffusion models have significantly facilitated text-guided video editing. However, there is a relative scarcity of research on image-guided video editing, a method that empowers users to edit videos by merely indicating a target object in the initial frame and providing an RGB image as reference, without relying on the text prompts. In this paper, we propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff for the image-guided video editing. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Inspired by self-supervised learning concepts, we introduce a masked motion modeling fine-tuning strategy that empowers the motion module's capabilities for capturing inter-frame motion dynamics, while preserving the capabilities for intra-frame semantic correlations modeling of the base image editing model. Moreover, an optical-flow-guided motion reference network is proposed to ensure the accurate propagation of information between edited video frames, alleviating the misleading effects of invalid information. We also construct a benchmark to facilitate further research. The comprehensive experiments demonstrate that our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。