让3D补全在任意视角下都保持几何一致与视觉真实。
IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement
- 用几何先验和多视角精修网络提升复杂场景的3D补全能力。
- 在自建多样化基准上，显著超越现有最佳方法。
- 适合需要真实感3D重建的工业级应用或虚拟现实开发。
当前的3D补全与物体移除方法主要局限于正面视角场景，在相机姿态和轨迹不受限的复杂场景中表现受限。为解决这一问题，我们提出一种新方法，可在正面及任意视角场景中生成视觉质量一致、几何结构连贯的3D补全结果。具体而言，我们构建了一个鲁棒的3D补全流程，融合几何先验与基于测试时适配训练的多视角精修网络，并依托预训练图像补全模型实现。此外，我们设计了一种新型掩码检测技术，可从物体掩码生成针对性补全掩码，显著提升对非约束场景的处理效果。为验证方法有效性，我们建立了一个涵盖多种场景的挑战性基准。大量实验表明，所提方法明显优于现有最先进方法。
原文摘要 · Abstract (English)
Current 3D inpainting and object removal methods are largely limited to front-facing scenes, facing substantial challenges when applied to diverse, "unconstrained" scenes where the camera orientation and trajectory are unrestricted. To bridge this gap, we introduce a novel approach that produces inpainted 3D scenes with consistent visual quality and coherent underlying geometry across both front-facing and unconstrained scenes. Specifically, we propose a robust 3D inpainting pipeline that incorporates geometric priors and a multi-view refinement network trained via test-time adaptation, building on a pre-trained image inpainting model. Additionally, we develop a novel inpainting mask detection technique to derive targeted inpainting masks from object masks, boosting the performance in handling unconstrained scenes. To validate the efficacy of our approach, we create a challenging and diverse benchmark that spans a wide range of scenes. Comprehensive experiments demonstrate that our proposed method substantially outperforms existing state-of-the-art approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。