arXiv:2409.13648cs.CVcs.GR2024-09被引 56

让手机流畅播放高质量3D视频,靠的是把3D动态高斯变成2D视频流。

V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians

  • 将动态3D高斯视为2D视频流,适配手机硬件编码器。
  • 两阶段训练减少存储量,实现快速压缩与高质量渲染。
  • 首个支持手机实时播放3D视频的系统,适合移动端用户体验。

在移动设备上像观看2D视频一样流畅体验高保真体素视频,是长久以来的梦想。然而,现有的动态3D高斯方法尽管渲染质量高,却因计算和带宽限制难以在移动设备上实现实时流媒体。本文提出V^3(Viewing Volumetric Videos),通过流式传输动态高斯实现高质量移动渲染。核心创新在于将动态3D高斯视为2D视频,从而利用硬件视频编解码器。我们设计了两阶段训练策略:第一阶段使用哈希编码和浅层MLP学习运动,通过剪枝减少高斯数量以满足流媒体要求;第二阶段使用残差熵损失和时间一致性损失微调其他高斯属性,提升时序连续性。该策略分离运动与外观,实现紧凑存储与高质量渲染。同时,我们开发了多平台播放器用于解码和渲染2D高斯视频。大量实验表明,V^3首次在普通设备上实现高质量3D视频流媒体,效果优于现有方法。项目主页提供源码与演示:https://authoritywang.github.io/v3/。

原文摘要 · Abstract (English)

Experiencing high-fidelity volumetric video as seamlessly as 2D videos is a long-held dream. However, current dynamic 3DGS methods, despite their high rendering quality, face challenges in streaming on mobile devices due to computational and bandwidth constraints. In this paper, we introduce V^3 (Viewing Volumetric Videos), a novel approach that enables high-quality mobile rendering through the streaming of dynamic Gaussians. Our key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of hardware video codecs. Additionally, we propose a two-stage training strategy to reduce storage requirements with rapid training speed. The first stage employs hash encoding and shallow MLP to learn motion, then reduces the number of Gaussians through pruning to meet the streaming requirements, while the second stage fine tunes other Gaussian attributes using residual entropy loss and temporal loss to improve temporal continuity. This strategy, which disentangles motion and appearance, maintains high rendering quality with compact storage requirements. Meanwhile, we designed a multi-platform player to decode and render 2D Gaussian videos. Extensive experiments demonstrate the effectiveness of V^3, outperforming other methods by enabling high-quality rendering and streaming on common devices, which is unseen before. As the first to stream dynamic Gaussians on mobile devices, our companion player offers users an unprecedented volumetric video experience, including smooth scrolling and instant sharing. Our project page with source code is available at https://authoritywang.github.io/v3/.

3D视频动态高斯移动端渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。