实时处理无相机参数的图像流,实现快速高保真3D重建。
StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams
- 逐帧预测并聚合高斯点,构建动态3D高斯流。
- 比传统方法快150倍,且在陌生场景下仍保持高精度。
- 适合需要即时反馈的AR/VR和机器人应用。
3D高斯泼溅(3DGS)技术推动了3D场景重建与新视角合成的发展。随着交互式应用对实时反馈需求增加,实时在线3DGS重建成为迫切需求。然而现有方法受限于三个挑战:缺乏预设相机参数、需具备泛化优化能力、以及冗余信息过多。本文提出StreamGS,一种针对无姿态图像流的在线泛化3DGS重建方法,通过预测并聚合每帧高斯点,逐步将图像流转化为3D高斯流。该方法克服了初始点重建在域外(OOD)场景下的局限性,引入内容自适应精修机制,通过建立相邻帧间可靠像素对应关系,增强跨帧一致性,并利用跨帧特征聚合合并冗余高斯点,显著降低密度。实验表明,StreamGS在多个数据集上达到与优化方法相当的重建质量,但速度提升150倍,且在处理域外场景时表现出更强泛化能力。
原文摘要 · Abstract (English)
The advent of 3D Gaussian Splatting (3DGS) has advanced 3D scene reconstruction and novel view synthesis. With the growing interest of interactive applications that need immediate feedback, online 3DGS reconstruction in real-time is in high demand. However, none of existing methods yet meet the demand due to three main challenges: the absence of predetermined camera parameters, the need for generalizable 3DGS optimization, and the necessity of reducing redundancy. We propose StreamGS, an online generalizable 3DGS reconstruction method for unposed image streams, which progressively transform image streams to 3D Gaussian streams by predicting and aggregating per-frame Gaussians. Our method overcomes the limitation of the initial point reconstruction \cite{dust3r} in tackling out-of-domain (OOD) issues by introducing a content adaptive refinement. The refinement enhances cross-frame consistency by establishing reliable pixel correspondences between adjacent frames. Such correspondences further aid in merging redundant Gaussians through cross-frame feature aggregation. The density of Gaussians is thereby reduced, empowering online reconstruction by significantly lowering computational and memory costs. Extensive experiments on diverse datasets have demonstrated that StreamGS achieves quality on par with optimization-based approaches but does so 150 times faster, and exhibits superior generalizability in handling OOD scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。