用单个双目图像实现高质量3D高斯点云重建,支持设备端实时推理。
StereoSplat+: Feed-Forward Stereo Gaussian Splatting with Diffusion-Assisted Progressive Inference

- 基于双目输入的前馈式3D高斯估计器,支持不同视角数量和相机配置
- 引入扩散增强的渐进推理机制,通过生成新视图反哺模型优化
- 适用于机器人、AR等单侧双目感知场景,尤其在遮挡区域表现优异
最近的3D高斯点云(3DGS)进展实现了高质量、可渲染的场景表示,用于新视角合成。然而,大多数现有3DGS流程依赖多视角观测(或非因果的未来帧访问),这在设备端机器人和增强现实(AR)场景中难以实现,因感知仅限于单一双目相机。因此,仅从一个双目观测中恢复高质量3DGS场景仍具挑战性,受限于遮挡、视野有限及几何缺失。本文提出StereoSplat+,一种结合扩散模型的前馈式双目3DGS框架,支持因果重建。方法包含两个关键组件:首先,提出StereoSplat,一种输入无关的前馈式3D高斯估计算法,能处理任意数量的带位姿双目图像对,并预测高质量3D高斯;其通过代价体积分支与基于三平面的3D体积分支融合互补几何线索,并利用连续位姿编码实现跨视角数量与相机配置的泛化。其次,由于推理时通常无法获取多个双目图像对,我们设计了名为StereoSplat+的扩散增强单次渐进推理方案:从一个双目对出发,根据预测的3DGS渲染新双目视图,经一步扩散增强后反馈作为额外输入以更新3DGS。KITTI-360数据集上的实验表明,StereoSplat+显著提升新视角渲染质量与几何准确性,特别是在遮挡区域和强视角外推情况下,优于近期前馈式3DGS基线。
原文摘要 · Abstract (English)
Recent advances in 3D Gaussian Splatting (3DGS) have enabled high-quality, render-ready scene representations for novel-view synthesis. However, most existing 3DGS pipelines rely on multi-view observations (or non-causal access to future frames) to achieve sufficient coverage, which is often unavailable in on-device robotics and AR settings where sensing is restricted to a single stereo rig. Recovering a high-quality 3DGS scene from one stereo observation, therefore, remains challenging due to occlusions, limited field of view, and missing geometry. We present StereoSplat+, a diffusion-enhanced feed-forward framework that enables causal reconstruction from a single stereo pair. Our method builds on two key components. First, we propose StereoSplat, an input-invariant feed-forward 3D Gaussian estimator that takes a variable number of posed stereo pairs as input and predicts high-quality 3D Gaussians. StereoSplat fuses complementary geometry cues via a cost-volume branch and a triplane-based 3D volume branch and leverages continuous pose encoding to generalize across view counts and camera configurations. Second, since multiple posed stereo pairs are typically unavailable at inference time, we introduce a diffusion-enhanced one-shot progressive inference scheme called StereoSplat+: starting from one stereo pair, we render novel stereo views from the predicted 3DGS, refine them with a one-step diffusion enhancer, and feed them back as additional inputs to update the 3DGS. Experiments on the KITTI-360 dataset show that StereoSplat+ improves novel-view rendering quality and geometry accuracy, especially in occluded regions and under strong view extrapolation, outperforming recent feed-forward 3DGS baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。