无需精确相机位姿，也能高效生成高质量3D新视角图像。
PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting
- 用预训练深度与视觉对应模型粗对齐3D高斯点，再轻量微调提升精度。
- 在真实世界数据集上优于现有方法，重建质量与视图合成效果均达新标杆。
- 适合无位姿或低重叠图像的3D重建任务，尤其适合实际应用部署。
我们研究从无位姿图像中实现单次前馈的新视角合成问题。该框架利用3D高斯溅射（3DGS）的快速、可扩展和高质量重建能力，进一步放宽了对密集图像视角、精确相机位姿和大量图像重叠等常见假设的要求。通过识别并解决像素对齐3DGS中的独特挑战——不同视角间3D高斯点错位导致梯度噪声或稀疏，从而影响训练稳定性和收敛性——我们采用预训练的单目深度估计和视觉对应模型实现3D高斯点的粗对齐。随后引入轻量级可学习模块，从粗对齐中优化深度与位姿估计，提升3D重建与新视角合成质量。此外，利用优化后的估计结果计算几何置信度分数，评估3D高斯中心可靠性，并据此调节高斯参数预测。在大规模真实世界数据集上的广泛评估表明，PF3plat 在所有基准上均达到新最优性能，且通过详尽消融实验验证了设计选择的有效性。
原文摘要 · Abstract (English)
We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate this, we employ pre-trained monocular depth estimation and visual correspondence models to achieve coarse alignments of 3D Gaussians. We then introduce lightweight, learnable modules to refine depth and pose estimates from the coarse alignments, improving the quality of 3D reconstruction and novel view synthesis. Furthermore, the refined estimates are leveraged to estimate geometry confidence scores, which assess the reliability of 3D Gaussian centers and condition the prediction of Gaussian parameters accordingly. Extensive evaluations on large-scale real-world datasets demonstrate that PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. project page: https://cvlab-kaist.github.io/PF3plat/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。