用扩散模型实现动画帧插值，能自适应时间并估计不确定性。
Time-adaptive Video Frame Interpolation based on Residual Diffusion
- 显式建模插值时间并训练中重估，适配动画域大时间差异。
- 仅需约10次扩散步骤即可生成高质量插值帧。
- 利用随机性输出像素级置信度，帮助识别错误区域。
本文提出一种基于扩散模型的视频帧插值方法，针对传统手绘动画场景。主要贡献有三：首先，显式建模插值时间，并在训练中动态重估，以应对动画领域相较于自然视频更大的时间变化；其次，将超分辨率领域提出的ResShift扩散框架适配并推广至视频插值，实现仅需约10步扩散即可生成结果；最后，利用扩散过程的随机性，提供像素级插值结果不确定性估计，有助于预测模型可能出错的位置。我们在多个动画视频数据集上与当前最优模型对比，结果表明本方法性能更优。代码已公开于https://github.com/VicFonch/Multi-Input-Resshift-Diffusion-VFI。
原文摘要 · Abstract (English)
In this work, we propose a new diffusion-based method for video frame interpolation (VFI), in the context of traditional hand-made animation. We introduce three main contributions: The first is that we explicitly handle the interpolation time in our model, which we also re-estimate during the training process, to cope with the particularly large variations observed in the animation domain, compared to natural videos; The second is that we adapt and generalize a diffusion scheme called ResShift recently proposed in the super-resolution community to VFI, which allows us to perform a very low number of diffusion steps (in the order of 10) to produce our estimates; The third is that we leverage the stochastic nature of the diffusion process to provide a pixel-wise estimate of the uncertainty on the interpolated frame, which could be useful to anticipate where the model may be wrong. We provide extensive comparisons with respect to state-of-the-art models and show that our model outperforms these models on animation videos. Our code is available at https://github.com/VicFonch/Multi-Input-Resshift-Diffusion-VFI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。