通过概率融合光流与深度,提升稀疏视角3D重建精度。
JointSplat: Probabilistic Joint Flow-Depth Optimization for Sparse-View Gaussian Splatting
- 基于光流匹配概率动态融合深度与光流信息
- 在RealEstate10K和ACID上优于当前最佳方法
- 适合需要高保真稀疏视图3D重建的研究者
从稀疏视角重建3D场景是长期存在的挑战,具有广泛的应用前景。近期基于前向传播的3D高斯稀疏视图重建方法利用大规模多视角数据学习几何先验,并通过反投影计算3D高斯中心,实现了实时新视角合成。尽管提供了强几何线索,前向多视角深度估计与光流-深度联合估计仍存在关键局限:前者在低纹理或重复区域易出现位置错误和伪影,后者因缺乏真实光流监督而依赖不可靠匹配,导致局部噪声和全局不一致。为此,我们提出JointSplat,一种统一框架,通过新颖的概率优化机制,利用光流与深度的互补性。具体而言,该像素级机制在训练中根据光流匹配概率动态调整深度与光流的信息融合强度。在此基础上,我们进一步提出一种新型多视角深度一致性损失,以利用可靠监督信号,同时抑制不确定区域的误导梯度。在RealEstate10K和ACID数据集上的评估表明,JointSplat持续优于现有最先进方法,验证了所提概率联合光流-深度优化方法在高保真稀疏视图3D重建中的有效性与鲁棒性。
原文摘要 · Abstract (English)
Reconstructing 3D scenes from sparse viewpoints is a long-standing challenge with wide applications. Recent advances in feed-forward 3D Gaussian sparse-view reconstruction methods provide an efficient solution for real-time novel view synthesis by leveraging geometric priors learned from large-scale multi-view datasets and computing 3D Gaussian centers via back-projection. Despite offering strong geometric cues, both feed-forward multi-view depth estimation and flow-depth joint estimation face key limitations: the former suffers from mislocation and artifact issues in low-texture or repetitive regions, while the latter is prone to local noise and global inconsistency due to unreliable matches when ground-truth flow supervision is unavailable. To overcome this, we propose JointSplat, a unified framework that leverages the complementarity between optical flow and depth via a novel probabilistic optimization mechanism. Specifically, this pixel-level mechanism scales the information fusion between depth and flow based on the matching probability of optical flow during training. Building upon the above mechanism, we further propose a novel multi-view depth-consistency loss to leverage the reliability of supervision while suppressing misleading gradients in uncertain areas. Evaluated on RealEstate10K and ACID, JointSplat consistently outperforms state-of-the-art (SOTA) methods, demonstrating the effectiveness and robustness of our proposed probabilistic joint flow-depth optimization approach for high-fidelity sparse-view 3D reconstruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。