用2D高斯点图高效表示视频，内存减少78.4%。
GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting
- 基于可变形2D高斯点图动态建模帧间变化
- 相比最新NeRV方法，训练快5.5倍，解码快12.5倍
- 适合追求高效视频压缩与实时处理的开发者
隐式神经视频表示（NeRV）为视频表示与压缩带来新范式，超越传统编码器。但模型规模增大时，编码解码速度慢、内存消耗高，限制实际应用。为此，我们提出基于2D高斯点图的新视频表示与压缩方法。通过可变形2D高斯点图在每帧动态调整变换，显著降低内存开销。结合多平面时空编码器与轻量解码器，模型根据时间步预测初始高斯点的颜色、坐标和形状变化。利用时间梯度有效捕捉时间冗余，代价极低。实验表明，该方法最多降低78.4%的GPU内存占用，并使视频处理速度大幅提升：训练效率提升5.5倍，解码速度提升12.5倍，优于当前最优的NeRV方法。
原文摘要 · Abstract (English)
Implicit Neural Representation for Videos (NeRV) has introduced a novel paradigm for video representation and compression, outperforming traditional codecs. As model size grows, however, slow encoding and decoding speed and high memory consumption hinder its application in practice. To address these limitations, we propose a new video representation and compression method based on 2D Gaussian Splatting to efficiently handle video data. Our proposed deformable 2D Gaussian Splatting dynamically adapts the transformation of 2D Gaussians at each frame, significantly reducing memory cost. Equipped with a multi-plane-based spatiotemporal encoder and a lightweight decoder, it predicts changes in color, coordinates, and shape of initialized Gaussians, given the time step. By leveraging temporal gradients, our model effectively captures temporal redundancy at negligible cost, significantly enhancing video representation efficiency. Our method reduces GPU memory usage by up to 78.4%, and significantly expedites video processing, achieving 5.5x faster training and 12.5x faster decoding compared to the state-of-the-art NeRV methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。