用轻量模型实现低延迟视频预测，提升实时交互体验。
Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training
- 基于IFRNet改进的卷积网络，结合三种新训练方法提升预测能力。
- 引入ELAN残差块，使模型在速度与精度间达到最优平衡。
- 适合需要低延迟视频传输的实时应用，如远程控制、在线会议。
传输延迟显著影响实时交互和操作的用户体验。由于延迟难以完全避免，视频预测可用来缓解延迟问题，最终实现零延迟传输。然而，现有大多数视频预测方法计算开销大，不适用于实时场景。为此，本文提出面向零延迟网络交互的实时视频预测方法——IFRVP（Intermediate Feature Refinement Video Prediction）。首先，我们基于IFRNet提出一种仅含卷积的轻量级帧插值网络，并设计三种扩展的视频预测训练方法；其次，引入基于ELAN的残差块以同时提升推理速度与精度。实验表明，所提模型在预测精度与计算速度之间实现了最佳权衡，优于现有方法。演示视频见 http://bit.ly/IFRVPDemo，代码将发布于 https://github.com/FykAikawa/IFRVP。
原文摘要 · Abstract (English)
Transmission latency significantly affects users' quality of experience in real-time interaction and actuation. As latency is principally inevitable, video prediction can be utilized to mitigate the latency and ultimately enable zero-latency transmission. However, most of the existing video prediction methods are computationally expensive and impractical for real-time applications. In this work, we therefore propose real-time video prediction towards the zero-latency interaction over networks, called IFRVP (Intermediate Feature Refinement Video Prediction). Firstly, we propose three training methods for video prediction that extend frame interpolation models, where we utilize a simple convolution-only frame interpolation network based on IFRNet. Secondly, we introduce ELAN-based residual blocks into the prediction models to improve both inference speed and accuracy. Our evaluations show that our proposed models perform efficiently and achieve the best trade-off between prediction accuracy and computational speed among the existing video prediction methods. A demonstration movie is also provided at http://bit.ly/IFRVPDemo. The code will be released at https://github.com/FykAikawa/IFRVP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。