统一脉冲图像重建、位姿校正与高斯溅射,端到端提升3D重建精度
USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting
- 将脉冲图像重建、位姿优化与高斯溅射联合优化,避免级联误差累积
- 在合成数据上相比之前方法显著提升3D重建质量,细节更清晰
- 适用于初始位姿不准的真实场景,有效降噪并保留纹理细节
脉冲相机以每秒40 kHz的0-1比特流捕捉场景,正被广泛用于基于神经辐射场(NeRF)或3D高斯溅射(3DGS)的3D重建任务。现有方法多采用级联流程:先用成熟算法从脉冲流重建高质量图像,再进行位姿估计和3D重建。但该流程存在显著的累积误差,初始图像重建质量差会直接影响位姿估计,最终降低3D重建保真度。为此,我们提出协同优化框架USP-Gaussian,将脉冲图像重建、位姿校正与高斯溅射统一为端到端流程。利用3DGS的多视图一致性及脉冲相机的运动捕捉能力,实现脉冲到图像网络与3DGS间的信息无缝融合与迭代优化。在具有准确位姿的合成数据集上实验表明,本方法有效消除级联误差,性能超越先前方法。进一步引入位姿优化,在真实场景中初始位姿不准确时仍能实现鲁棒3D重建,显著降低噪声并保持精细纹理。代码、数据与训练模型将公开于https://github.com/chenkang455/USP-Gaussian。
原文摘要 · Abstract (English)
Spike cameras, as an innovative neuromorphic camera that captures scenes with the 0-1 bit stream at 40 kHz, are increasingly employed for the 3D reconstruction task via Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS). Previous spike-based 3D reconstruction approaches often employ a casecased pipeline: starting with high-quality image reconstruction from spike streams based on established spike-to-image reconstruction algorithms, then progressing to camera pose estimation and 3D reconstruction. However, this cascaded approach suffers from substantial cumulative errors, where quality limitations of initial image reconstructions negatively impact pose estimation, ultimately degrading the fidelity of the 3D reconstruction. To address these issues, we propose a synergistic optimization framework, \textbf{USP-Gaussian}, that unifies spike-based image reconstruction, pose correction, and Gaussian splatting into an end-to-end framework. Leveraging the multi-view consistency afforded by 3DGS and the motion capture capability of the spike camera, our framework enables a joint iterative optimization that seamlessly integrates information between the spike-to-image network and 3DGS. Experiments on synthetic datasets with accurate poses demonstrate that our method surpasses previous approaches by effectively eliminating cascading errors. Moreover, we integrate pose optimization to achieve robust 3D reconstruction in real-world scenarios with inaccurate initial poses, outperforming alternative methods by effectively reducing noise and preserving fine texture details. Our code, data and trained models will be available at https://github.com/chenkang455/USP-Gaussian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。