arXiv:2412.11755cs.CV2024-12CVPR被引 28

用逐帧条件驱动生成中间帧,解决长距离运动插值不稳问题。

Generative Inbetweening through Frame-wise Conditions-Driven Video Generation

  • 为每帧添加匹配线作为显式条件,明确插值路径。
  • 在自然场景、人体姿态等复杂场景中实现稳定视频生成。
  • 支持线性与非线性插值,适合动画和视频修复任务。

生成式插帧旨在利用两个关键帧生成中间帧序列。尽管视频生成模型取得显著进展,但生成式插帧仍面临时间稳定性不足的问题,尤其在输入帧间运动差距较大时更为严重。本文提出一种简单而高效的逐帧条件驱动视频生成(FCVG)方法,显著提升插值视频的时间稳定性。具体而言,FCVG为每一帧提供显式条件,使两关键帧间的插值路径更易识别,从而保证视觉合理且时间稳定的视频帧生成。为此,我们从两输入帧中提取匹配线,逐帧进行插值,并将其无缝集成至现有视频生成模型中。在涵盖自然景观、复杂人体姿态、摄像机运动及动画等多种场景的广泛评估中,现有方法常出现帧间不连贯现象。相比之下,我们的FCVG能使用线性和非线性插值曲线生成时间稳定视频。项目页面与代码见https://fcvg-inbetween.github.io/。

原文摘要 · Abstract (English)

Generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input. Although remarkable progress has been made in video generation models, generative inbetweening still faces challenges in maintaining temporal stability due to the ambiguous interpolation path between two key frames. This issue becomes particularly severe when there is a large motion gap between input frames. In this paper, we propose a straightforward yet highly effective Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames. Specifically, our FCVG provides an explicit condition for each frame, making it much easier to identify the interpolation path between two input frames and thus ensuring temporally stable production of visually plausible video frames. To achieve this, we suggest extracting matched lines from two input frames that can then be easily interpolated frame by frame, serving as frame-wise conditions seamlessly integrated into existing video generation models. In extensive evaluations covering diverse scenarios such as natural landscapes, complex human poses, camera movements and animations, existing methods often exhibit incoherent transitions across frames. In contrast, our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear interpolation curves. Our project page and code are available at \url{https://fcvg-inbetween.github.io/}.

视频生成插帧时序稳定条件生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。