用2D高斯表示视频,800+帧率下仍保35+分,训练仅需2秒/帧。
GSVR: 2D Gaussian-based Video Representation for 800+ FPS with Hybrid Deformation Field
- 用混合形变场建模视频动态,融合三平面与多项式运动模式。
- 在Bunny数据集上实现35+ PSNR、800+ FPS,训练仅2秒/帧。
- 适合需要高速解码与压缩的实时视频应用,如VR/AR传输。
隐式神经视频表示作为一种新型视频表达方式备受关注,但现有方法多聚焦重建质量,忽视解码速度。传统基于卷积网络的方法计算量大,导致解码慢,且训练耗时长,例如在Bunny数据集上每帧需14秒才能达到35+ PSNR。为此,本文提出GSVR——一种基于2D高斯的视频表示方法,可在Bunny上实现35+ PSNR和800+ FPS,每帧训练时间仅需2秒。核心创新包括:(1)提出混合形变场,结合三平面运动与多项式运动,有效处理相机与物体运动耦合;(2)设计动态感知的时间切片策略,根据视频动态水平自适应划分图像组(GOP),应对大范围相机运动与非刚性变形;(3)引入量化感知微调与图像编解码器压缩高斯分布,实现紧凑表示。在Bunny与UVG数据集上的实验表明,本方法收敛速度远超现有方法,解码速度比其他方法快10倍,视频插值任务性能接近当前最优,压缩表现优于NeRV。
原文摘要 · Abstract (English)
Implicit neural representations for video have been recognized as a novel and promising form of video representation. Existing works pay more attention to improving video reconstruction quality but little attention to the decoding speed. However, the high computation of convolutional network used in existing methods leads to low decoding speed. Moreover, these convolution-based video representation methods also suffer from long training time, about 14 seconds per frame to achieve 35+ PSNR on Bunny. To solve the above problems, we propose GSVR, a novel 2D Gaussian-based video representation, which achieves 800+ FPS and 35+ PSNR on Bunny, only needing a training time of $2$ seconds per frame. Specifically, we propose a hybrid deformation field to model the dynamics of the video, which combines two motion patterns, namely the tri-plane motion and the polynomial motion, to deal with the coupling of camera motion and object motion in the video. Furthermore, we propose a Dynamic-aware Time Slicing strategy to adaptively divide the video into multiple groups of pictures(GOP) based on the dynamic level of the video in order to handle large camera motion and non-rigid movements. Finally, we propose quantization-aware fine-tuning to avoid performance reduction after quantization and utilize image codecs to compress Gaussians to achieve a compact representation. Experiments on the Bunny and UVG datasets confirm that our method converges much faster than existing methods and also has 10x faster decoding speed compared to other methods. Our method has comparable performance in the video interpolation task to SOTA and attains better video compression performance than NeRV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。