arXiv:2502.04630cs.CVcs.GR2025-02被引 6

融合三种相机,实时重建高速动态3D场景

High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting

  • 用可变形3D高斯表示场景,融合RGB、深度和事件相机数据
  • 在低光、快速运动等挑战条件下仍保持高精度重建
  • 适合机器人视觉、生物运动分析等高速动态场景应用

高速动态3D场景的捕获与重建在计算机图形学、视觉及机器人、空气动力学、进化生物学等跨学科领域具有广泛应用。然而,仅靠单一成像模态难以实现。传统RGB相机存在帧率低、曝光时间短、基线窄等问题。为此,本文提出一种基于高斯点阵的传感器融合新方法,结合RGB、深度和事件相机,实现高速动态变形场景的捕捉与重建。核心思想是利用三类相机的互补优势:RGB相机提供细节颜色信息,事件相机以微秒级分辨率记录快速变化,深度相机提供三维几何结构。为统一多模态的场景表达,采用可变形3D高斯建模,并通过联合优化3D高斯参数及其时序形变场,融合三源数据。该方法在合成与真实数据集上均显著优于现有技术,在渲染保真度和结构准确性方面均有明显提升,即使在低光、窄基线或快速运动等复杂条件下依然表现优异。

原文摘要 · Abstract (English)

Capturing and reconstructing high-speed dynamic 3D scenes has numerous applications in computer graphics, vision, and interdisciplinary fields such as robotics, aerodynamics, and evolutionary biology. However, achieving this using a single imaging modality remains challenging. For instance, traditional RGB cameras suffer from low frame rates, limited exposure times, and narrow baselines. To address this, we propose a novel sensor fusion approach using Gaussian splatting, which combines RGB, depth, and event cameras to capture and reconstruct deforming scenes at high speeds. The key insight of our method lies in leveraging the complementary strengths of these imaging modalities: RGB cameras capture detailed color information, event cameras record rapid scene changes with microsecond resolution, and depth cameras provide 3D scene geometry. To unify the underlying scene representation across these modalities, we represent the scene using deformable 3D Gaussians. To handle rapid scene movements, we jointly optimize the 3D Gaussian parameters and their temporal deformation fields by integrating data from all three sensor modalities. This fusion enables efficient, high-quality imaging of fast and complex scenes, even under challenging conditions such as low light, narrow baselines, or rapid motion. Experiments on synthetic and real datasets captured with our prototype sensor fusion setup demonstrate that our method significantly outperforms state-of-the-art techniques, achieving noticeable improvements in both rendering fidelity and structural accuracy.

3D重建传感器融合高斯点阵高速动态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。