arXiv:2501.03699cs.CV2025-01被引 18

融合运动流与生成能力,提升复杂运动场景的帧插值质量

Motion-Aware Generative Frame Interpolation

  • 用运动流引导生成过程,双层级注入条件信息
  • 在真实世界和动画数据集上均超越现有方法
  • 适合需要高稳定性和细节保真的视频插帧场景

基于光流的帧插值方法通过估计中间光流保证运动稳定性,但在复杂运动区域常引入严重伪影。近期借助大规模预训练视频生成模型的生成方法在处理复杂场景方面展现出潜力,但因缺乏显式的运动轨迹约束,常导致运动不稳和内容不一致。为此,我们提出运动感知生成式帧插值(MoG),将中间光流引导与生成能力相结合,以提升插值保真度。核心思想是在光流约束下保持运动平滑的同时,通过生成式修正光流估计误差。具体而言,我们引入双层引导注入机制,在潜在空间和特征空间同时利用中间光流传播条件信息,使生成运动与光流推导的运动轨迹对齐;同时设计编码器仅引导注入与选择性参数微调,实现复杂运动区域的动态伪影修正。在真实世界和动画基准上的大量实验表明,MoG 在视频质量和视觉保真度方面均优于当前最优方法。本工作弥合了基于光流的稳定性与生成灵活性之间的差距,为多样场景下的帧插值提供通用解决方案。

原文摘要 · Abstract (English)

Flow-based frame interpolation methods ensure motion stability through estimated intermediate flow but often introduce severe artifacts in complex motion regions. Recent generative approaches, boosted by large-scale pre-trained video generation models, show promise in handling intricate scenes. However, they frequently produce unstable motion and content inconsistencies due to the absence of explicit motion trajectory constraints. To address these challenges, we propose Motion-aware Generative frame interpolation (MoG) that synergizes intermediate flow guidance with generative capacities to enhance interpolation fidelity. Our key insight is to simultaneously enforce motion smoothness through flow constraints while adaptively correcting flow estimation errors through generative refinement. Specifically, we first introduce a dual guidance injection that propagates condition information using intermediate flow at both latent and feature levels, aligning the generated motion with flow-derived motion trajectories. Meanwhile, we implemented two critical designs, encoder-only guidance injection and selective parameter fine-tuning, which enable dynamic artifact correction in the complex motion regions. Extensive experiments on both real-world and animation benchmarks demonstrate that MoG outperforms state-of-the-art methods in terms of video quality and visual fidelity. Our work bridges the gap between flow-based stability and generative flexibility, offering a versatile solution for frame interpolation across diverse scenarios.

帧插值生成模型运动建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。