提出一种高效视频神经表示,编码速度提升3倍以上且参数更少。
Efficient Neural Video Representation with Temporally Coherent Modulation
- 通过流信息分解视频为2D网格,实现时序一致的动态建模。
- 在相同质量下比传统网格方法快3倍以上,参数减少10%仍保持更高精度。
- 适用于超分、插帧、补全等任务,压缩性能媲美H.264/HEVC。
隐式神经表示(INR)已在多个领域取得成功。为推动其在真实场景中的应用,加速训练至关重要。现有基于网格的视频INR方法虽提升了编码速度,但因忽略视频动态特性,导致参数冗余,效率低于不使用参数编码的NeRV方法。为此,我们提出时空一致调制神经视频表示(NVTM),将时空3D视频数据分解为带运动信息的2D网格,实现快速学习与高效参数利用。该框架可一次性处理时序对应像素,使编码速度在合理质量下达到最快,较NeRV方法提速超过3倍。在UVG(动态)上平均提升PSNR 1.54dB、LPIPS 0.019(参数少10%);在MCL-JCV(动态)上平均提升PSNR 1.84dB、LPIPS 0.013。扩展至压缩任务后,性能接近H.264、HEVC及最新INR压缩方法。大量实验验证了其在超分辨率、帧插值和视频修复等任务中的优越性。
原文摘要 · Abstract (English)
Implicit neural representations (INR) has found successful applications across diverse domains. To employ INR in real-life, it is important to speed up training. In the field of INR for video applications, the state-of-the-art approach employs grid-type parametric encoding and successfully achieves a faster encoding speed in comparison to its predecessors. However, the grid usage, which does not consider the video's dynamic nature, leads to redundant use of trainable parameters. As a result, it has significantly lower parameter efficiency and higher bitrate compared to NeRV-style methods that do not use a parametric encoding. To address the problem, we propose Neural Video representation with Temporally coherent Modulation (NVTM), a novel framework that can capture dynamic characteristics of video. By decomposing the spatio-temporal 3D video data into a set of 2D grids with flow information, NVTM enables learning video representation rapidly and uses parameter efficiently. Our framework enables to process temporally corresponding pixels at once, resulting in the fastest encoding speed for a reasonable video quality, especially when compared to the NeRV-style method, with a speed increase of over 3 times. Also, it remarks an average of 1.54dB/0.019 improvements in PSNR/LPIPS on UVG (Dynamic) (even with 10% fewer parameters) and an average of 1.84dB/0.013 improvements in PSNR/LPIPS on MCL-JCV (Dynamic), compared to previous grid-type works. By expanding this to compression tasks, we demonstrate comparable performance to video compression standards (H.264, HEVC) and recent INR approaches for video compression. Additionally, we perform extensive experiments demonstrating the superior performance of our algorithm across diverse tasks, encompassing super resolution, frame interpolation and video inpainting. Project page is https://sujiikim.github.io/NVTM/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。