通过多视角实例对齐,提升单目深度先验的准确性。
MonoInstance: Enhancing Monocular Priors via Multi-view Instance Alignment for Neural Rendering and Reconstruction
- 将多视角实例深度对齐到统一3D空间,生成更可靠的几何先验。
- 在高不确定性区域引入投影对齐约束,减少错误深度影响。
- 可无缝集成到多种神经渲染框架,适合3D重建与新视角合成任务。
单目深度先验被广泛应用于基于多视角的神经渲染任务,如3D重建和新视角合成。然而,由于各视角间预测不一致,如何有效利用单目线索仍具挑战。现有方法将整个估计的深度图当作真实标签进行监督,忽视了单目先验中固有的不准确性和跨视角不一致性。为此,我们提出MonoInstance,一种通用方法,通过探索单目深度的不确定性,为神经渲染与重建提供增强的几何先验。核心思想是将多视角中分割出的每个实例深度在共同3D空间内对齐,从而将单目深度的不确定性转化为噪声点云中的密度度量。对于深度先验不可靠的高不确定性区域,我们进一步引入约束项,促使投影实例与邻近视图对应的实例掩码对齐。MonoInstance是一种通用策略,可无缝集成至多种多视角神经渲染框架中。实验结果表明,该方法在多个基准测试中显著提升了重建与新视角合成性能。
原文摘要 · Abstract (English)
Monocular depth priors have been widely adopted by neural rendering in multi-view based tasks such as 3D reconstruction and novel view synthesis. However, due to the inconsistent prediction on each view, how to more effectively leverage monocular cues in a multi-view context remains a challenge. Current methods treat the entire estimated depth map indiscriminately, and use it as ground truth supervision, while ignoring the inherent inaccuracy and cross-view inconsistency in monocular priors. To resolve these issues, we propose MonoInstance, a general approach that explores the uncertainty of monocular depths to provide enhanced geometric priors for neural rendering and reconstruction. Our key insight lies in aligning each segmented instance depths from multiple views within a common 3D space, thereby casting the uncertainty estimation of monocular depths into a density measure within noisy point clouds. For high-uncertainty areas where depth priors are unreliable, we further introduce a constraint term that encourages the projected instances to align with corresponding instance masks on nearby views. MonoInstance is a versatile strategy which can be seamlessly integrated into various multi-view neural rendering frameworks. Our experimental results demonstrate that MonoInstance significantly improves the performance in both reconstruction and novel view synthesis under various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。