无需相机姿态,仅用零散无标注图像即可实时重建3D高斯场景。
No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images
- 用单个视图做坐标基准,直接预测所有视角的3D高斯参数。
- 在低重叠图像下,新视角合成质量优于依赖姿态的方法。
- 训练时不用真实深度或匹配损失,仍可精准估计相机姿态。
我们提出NoPoSplat,一种前馈模型,仅需稀疏多视角无标注图像即可重建3D高斯参数化场景。模型仅使用光度损失训练,推理时实现实时3D高斯重建。通过将输入中某一视图的本地相机坐标设为标准空间,网络可直接预测所有视图在此空间内的高斯原型,避免了逐帧转换至全局坐标带来的误差。为解决尺度歧义,设计并比较多种内参嵌入方法,最终选择将相机内参转为标记嵌入,并与图像标记拼接输入模型,实现精确尺度预测。利用重建的3D高斯进行新视角合成与位姿估计,提出两阶段粗到精的位姿估计流程。实验表明,该无姿态方法在输入图像重叠较少时,新视角合成效果优于依赖姿态的方法;在位姿估计任务中,未使用真值深度或显式匹配损失,仍显著超越当前最优方法。本工作推动了无姿态通用3D重建的发展,并验证其在真实场景中的适用性。代码与模型见https://noposplat.github.io/。
原文摘要 · Abstract (English)
We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from \textit{unposed} sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available at https://noposplat.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。