用残差流扩散模型实现高效可变长视频逐帧编辑
RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing
- 基于图像扩散模型改造，通过前一帧预测引导当前帧编辑
- 在保持图像级计算量下，支持任意长度视频编辑且速度不随长度增加而下降
- 新提出基准评测方法，适用于全局/局部风格迁移与物体移除任务
指令式视频编辑仅用文本提示即可对输入视频进行修改，实现自然语言控制。尽管进展迅速，多数方法仍需固定长度输入且计算开销大。自回归视频生成虽能高效处理可变长度内容，但在视频编辑中应用较少。本文提出一种因果性、高效的视频编辑模型，可逐帧编辑可变长度视频。为提升效率，从2D图像到图像（I2I）扩散模型出发，通过将时间步t的编辑条件设定为前一时刻t-1的模型输出，实现视频到视频（V2V）编辑。为利用视频的时间冗余性，提出新的I2I扩散前向过程，促使模型预测目标输出与前一预测之间的残差。该方法称为残差流扩散模型（RFDM），使去噪过程聚焦于连续帧间的差异。此外，提出一个新基准以更准确评估现有编辑方法。在配对视频数据上训练用于全局/局部风格迁移和物体移除，RFDM性能优于I2I基线，媲美全时空（3D）V2V模型，同时计算量与图像模型相当，且不随输入视频长度增长而增加。
原文摘要 · Abstract (English)
Instructional video editing applies edits to an input video using only text prompts, enabling intuitive natural-language control. Despite rapid progress, most methods still require fixed-length inputs and substantial compute. Meanwhile, autoregressive video generation enables efficient variable-length synthesis, yet remains under-explored for video editing. We introduce a causal, efficient video editing model that edits variable-length videos frame by frame. For efficiency, we start from a 2D image-to-image (I2I) diffusion model and adapt it to video-to-video (V2V) editing by conditioning the edit at time step t on the model's prediction at t-1. To leverage videos' temporal redundancy, we propose a new I2I diffusion forward process formulation that encourages the model to predict the residual between the target output and the previous prediction. We call this Residual Flow Diffusion Model (RFDM), which focuses the denoising process on changes between consecutive frames. Moreover, we propose a new benchmark that better ranks state-of-the-art methods for editing tasks. Trained on paired video data for global/local style transfer and object removal, RFDM surpasses I2I-based methods and competes with fully spatiotemporal (3D) V2V models, while matching the compute of image models and scaling independently of input video length. More content can be found in: https://smsd75.github.io/RFDM_page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。