用自监督扩散模型实现低资源视频插帧,无需光流或高帧率数据。
MiVID: Multi-Strategic Self-Supervision for Video Frame Interpolation using Diffusion Model
- 结合3D U-Net与时间注意力,通过渐进掩码训练学习时空特征。
- 在UCF101-7和DAVIS-7上仅用50轮训练即达最优性能。
- 全程仅需CPU运行,适合资源受限场景下的视频增强应用。
视频帧插值(VFI)是视频增强的核心技术,支持慢动作生成、帧率转换与视频修复等任务。传统方法依赖光流或需要密集真值标签,难以应对遮挡、域偏移和运动模糊等问题。本文提出MiVID,一种轻量级、自监督的扩散模型框架,无需显式运动估计。模型采用3D U-Net主干网络与类Transformer的时间注意力机制,通过混合掩码策略模拟遮挡与运动不确定性。利用基于余弦的渐进掩码和自适应损失调度,网络在无高帧率监督下学习鲁棒的时空表征。在UCF101-7和DAVIS-7数据集上评估,模型仅用9帧视频片段和CPU训练50轮即达最优效果,性能媲美多个有监督基线。本工作展示了自监督扩散先验在时序一致帧合成中的潜力,为可扩展、泛化性强的VFI系统提供了新路径。
原文摘要 · Abstract (English)
Video Frame Interpolation (VFI) remains a cornerstone in video enhancement, enabling temporal upscaling for tasks like slow-motion rendering, frame rate conversion, and video restoration. While classical methods rely on optical flow and learning-based models assume access to dense ground-truth, both struggle with occlusions, domain shifts, and ambiguous motion. This article introduces MiVID, a lightweight, self-supervised, diffusion-based framework for video interpolation. Our model eliminates the need for explicit motion estimation by combining a 3D U-Net backbone with transformer-style temporal attention, trained under a hybrid masking regime that simulates occlusions and motion uncertainty. The use of cosine-based progressive masking and adaptive loss scheduling allows our network to learn robust spatiotemporal representations without any high-frame-rate supervision. Our framework is evaluated on UCF101-7 and DAVIS-7 datasets. MiVID is trained entirely on CPU using the datasets and 9-frame video segments, making it a low-resource yet highly effective pipeline. Despite these constraints, our model achieves optimal results at just 50 epochs, competitive with several supervised baselines.This work demonstrates the power of self-supervised diffusion priors for temporally coherent frame synthesis and provides a scalable path toward accessible and generalizable VFI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。