arXiv:2411.07541cs.CV2024-11中稿 · NeurIPS被引 5

提出高效动态场景重建框架,实时生成高质量视频并大幅节省存储。

HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting

  • 分层运动机制利用高斯点局部一致性,快速准确学习帧间运动。
  • 训练效率提升20%,存储量减少85%,每帧平均训练时间小于2秒。
  • 适合需要低延迟、高稳定性的实时3D场景重建应用。

从多视角流式视频中在线重建动态场景面临训练、渲染和存储效率的挑战。尽管3D高斯点云(3DGS)具备快速学习和实时渲染的优势,但在视图有限时仍存在存储效率低和过拟合问题。本文提出高效框架HiCoM,包含三个关键组件:首先,采用扰动平滑策略构建紧凑且鲁棒的初始3DGS表示;其次,引入分层一致运动机制,利用3D高斯点的非均匀分布与局部一致性,实现跨帧运动的快速精准建模;最后,通过持续添加并合并新高斯点来更新3DGS,同时移除等量低透明度高斯点以保持表示紧凑。在两个主流数据集上的实验表明,该方法使当前最优方法的学习效率提升约20%,数据存储减少85%,达到具有竞争力的自由视角视频合成质量,且更具鲁棒性与稳定性。通过并行处理多帧,平均每帧训练时间低于2秒,性能损失可忽略,显著提升实际应用中的响应速度与可用性。

原文摘要 · Abstract (English)

The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Harnessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient in terms of storage and prone to overfitting by excessively growing Gaussians, particularly with limited views. This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians to swiftly and accurately learn motions across frames. Finally, we continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve a compact representation, an equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about $20\%$ and reduces the data storage by $85\%$, achieving competitive free-viewpoint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to $<2$ seconds per frame with negligible performance degradation, substantially boosting real-world applicability and responsiveness.

3D重建动态场景实时渲染高斯点云

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。