通过分段处理与运动感知跳过，实现长视频高效编辑。
PipeFlow: Pipelined Processing and Motion-Aware Frame Selection for Long-Form Video Editing
- 按运动程度跳过低动量帧，减少冗余计算。
- 分段并行处理，编辑时间随视频长度线性增长。
- 用神经网络插值修复分段边界，适合超长视频编辑。
长视频编辑因在长序列上联合执行去噪扩散隐式模型（DDIM）反演而面临计算成本指数级上升的挑战。为此，我们提出PipeFlow，一种可扩展的流水线式视频编辑方法，包含三项关键创新：首先，基于结构相似性指数（SSIM）和光流分析运动程度，识别并跳过低运动帧的编辑；其次，设计流水线任务调度算法，将视频分段，根据可用显存并行执行DDIM反演与联合编辑；最后，采用基于神经网络的插值技术平滑段间边界，并补全被跳过的帧。该方法通过分段策略使编辑时间随视频长度线性增长，在理论上可支持无限长视频编辑，避免其他方法面临的每帧开销递增问题。相比TokenFlow，PipeFlow最高提升9.6倍；相比扩散运动迁移（DMT），提速达31.7倍。
原文摘要 · Abstract (English)
Long-form video editing poses unique challenges due to the exponential increase in the computational cost from joint editing and Denoising Diffusion Implicit Models (DDIM) inversion across extended sequences. To address these limitations, we propose PipeFlow, a scalable, pipelined video editing method that introduces three key innovations: First, based on a motion analysis using Structural Similarity Index Measure (SSIM) and Optical Flow, we identify and propose to skip editing of frames with low motion. Second, we propose a pipelined task scheduling algorithm that splits a video into multiple segments and performs DDIM inversion and joint editing in parallel based on available GPU memory. Lastly, we leverage a neural network-based interpolation technique to smooth out the border frames between segments and interpolate the previously skipped frames. Our method uniquely scales to longer videos by dividing them into smaller segments, allowing PipeFlow's editing time to increase linearly with video length. In principle, this enables editing of infinitely long videos without the growing per-frame computational overhead encountered by other methods. PipeFlow achieves up to a 9.6X speedup compared to TokenFlow and a 31.7X speedup over Diffusion Motion Transfer (DMT).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。