arXiv:2505.09324cs.CVcs.AI2025-05被引 3

用2D高斯点阵实现高效神经视频压缩,支持实时解码。

Neural Video Compression using 2D Gaussian Splatting

  • 基于内容感知初始化与帧间冗余消除,加速编码过程。
  • 编码速度提升88%,仅需数千个高斯点即可输出高质量视频。
  • 首个基于2D高斯点阵的神经视频压缩方案,适合实时会议应用。

计算机视觉与图像处理领域长期致力于视频数据通信的标准制定,如AVC、HEVC、VVC、AV1、AV2等。近年来,深度学习技术开始取代传统视频编解码器流程,神经视频编解码器(NVC)实现了端到端的机器学习解决方案,无需依赖人工设计的运动或边缘特征,能学习内容自适应的压缩策略,具有更高的适应性与压缩效率。这不仅有助于硬件设计,也适用于视频流媒体平台及视频会议(如MS-Teams、Zoom),在教学与办公场景中广泛应用。然而,其高计算需求限制了在实时场景中的使用。为此,本文提出一种基于兴趣区域(ROI)的神经视频压缩模型,利用2D高斯点阵。与传统编解码器不同,2D高斯点阵支持实时解码,并可通过少量数据点优化,仅需数千个高斯点即可生成良好质量输出,远低于3D场景中数百万的数量。本工作通过内容感知初始化策略和新颖的高斯帧间冗余减少机制,使基于高斯点阵的图像编解码器编码时间缩短88%,首次实现将高斯点阵应用于神经视频编解码器,为该领域提供新范式。

原文摘要 · Abstract (English)

The computer vision and image processing research community has been involved in standardizing video data communications for the past many decades, leading to standards such as AVC, HEVC, VVC, AV1, AV2, etc. However, recent groundbreaking works have focused on employing deep learning-based techniques to replace the traditional video codec pipeline to a greater affect. Neural video codecs (NVC) create an end-to-end ML-based solution that does not rely on any handcrafted features (motion or edge-based) and have the ability to learn content-aware compression strategies, offering better adaptability and higher compression efficiency than traditional methods. This holds a great potential not only for hardware design, but also for various video streaming platforms and applications, especially video conferencing applications such as MS-Teams or Zoom that have found extensive usage in classrooms and workplaces. However, their high computational demands currently limit their use in real-time applications like video conferencing. To address this, we propose a region-of-interest (ROI) based neural video compression model that leverages 2D Gaussian Splatting. Unlike traditional codecs, 2D Gaussian Splatting is capable of real-time decoding and can be optimized using fewer data points, requiring only thousands of Gaussians for decent quality outputs as opposed to millions in 3D scenes. In this work, we designed a video pipeline that speeds up the encoding time of the previous Gaussian splatting-based image codec by 88% by using a content-aware initialization strategy paired with a novel Gaussian inter-frame redundancy-reduction mechanism, enabling Gaussian splatting to be used for a video-codec solution, the first of its kind solution in this neural video codec space.

神经视频编码2D高斯点阵实时压缩视频会议

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。