用压缩后的高斯网格实现实时自由视角视频流,帧大小仅170KB
StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video
- 将动态场景分解为静态高斯、时间特征与形变场,用图像和视频编码压缩
- 平均帧大小降至170KB,PSNR比现有方法提升1dB,支持自适应码率
- 适合实时视频传输、低带宽场景,无需额外训练即可适配网络状况
实时流式自由视角视频(FVV)仍面临训练、渲染与传输效率的挑战。尽管基于3D高斯溅射(3DGS)的方法在训练与渲染上取得突破,但每帧存储高达10MB,难以实现实时流传输。为此,本文提出新型表示方法StreamSTGS,利用规范3D高斯、时间特征与形变场表示动态场景。通过将规范高斯属性编码为2D图像、时间特征编码为视频,实现高压缩率。该设计支持实时流传输,并可基于网络状况自动调节码率,无需额外训练。此外,采用滑动窗口聚合邻近时间特征以学习局部运动,并引入基于Transformer的辅助训练模块学习全局运动。在多个FVV基准上,StreamSTGS性能优于当前最优方法:平均PSNR提升1dB,帧大小降低至170KB。代码已开源。
原文摘要 · Abstract (English)
Streaming free-viewpoint video~(FVV) in real-time still faces significant challenges, particularly in training, rendering, and transmission efficiency. Harnessing superior performance of 3D Gaussian Splatting~(3DGS), recent 3DGS-based FVV methods have achieved notable breakthroughs in both training and rendering. However, the storage requirements of these methods can reach up to $10$MB per frame, making stream FVV in real-time impossible. To address this problem, we propose a novel FVV representation, dubbed StreamSTGS, designed for real-time streaming. StreamSTGS represents a dynamic scene using canonical 3D Gaussians, temporal features, and a deformation field. For high compression efficiency, we encode canonical Gaussian attributes as 2D images and temporal features as a video. This design not only enables real-time streaming, but also inherently supports adaptive bitrate control based on network condition without any extra training. Moreover, we propose a sliding window scheme to aggregate adjacent temporal features to learn local motions, and then introduce a transformer-guided auxiliary training module to learn global motions. On diverse FVV benchmarks, StreamSTGS demonstrates competitive performance on all metrics compared to state-of-the-art methods. Notably, StreamSTGS increases the PSNR by an average of $1$dB while reducing the average frame size to just $170$KB. The code is publicly available on https://github.com/kkkzh/StreamSTGS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。