用扩散模型修复激光扫描缺失的建筑立面，提升3D建模精度
FacaDiffy: Inpainting Unseen Facade Parts Using Diffusion Models
- 基于激光点云和3D模型生成冲突图，再用定制扩散模型补全缺失区域
- 合成数据训练使模型在真实场景下检测率提升22%
- 适合需要高精度3D建筑重建的研究者与工程师
高细节语义3D建筑模型广泛应用于机器人、地理信息学和计算机视觉。构建此类模型的关键是利用2D冲突图识别建筑立面的开口位置，但实际中因激光扫描障碍常导致冲突图不完整。为此，我们提出FacaDiffy，一种通过个性化Stable Diffusion模型完成冲突图的新型修补方法。首先，我们设计确定性射线分析法，从已有3D建筑模型及对应激光点云生成2D冲突图；其次，利用个人化Stable Diffusion模型将未见立面物体补全至冲突图中。为弥补真实数据稀缺，我们还开发了可扩展的合成数据生成管道，使用随机城市模型生成器与标注立面图像生成合成冲突图。大量实验表明，FacaDiffy在冲突图补全任务上优于多种基线方法，应用该补全结果进行高清3D语义建筑重建时，检测率提升22%。代码已公开于GitHub：https://github.com/ThomasFroech/InpaintingofUnseenFacadeObjects
原文摘要 · Abstract (English)
High-detail semantic 3D building models are frequently utilized in robotics, geoinformatics, and computer vision. One key aspect of creating such models is employing 2D conflict maps that detect openings' locations in building facades. Yet, in reality, these maps are often incomplete due to obstacles encountered during laser scanning. To address this challenge, we introduce FacaDiffy, a novel method for inpainting unseen facade parts by completing conflict maps with a personalized Stable Diffusion model. Specifically, we first propose a deterministic ray analysis approach to derive 2D conflict maps from existing 3D building models and corresponding laser scanning point clouds. Furthermore, we facilitate the inpainting of unseen facade objects into these 2D conflict maps by leveraging the potential of personalizing a Stable Diffusion model. To complement the scarcity of real-world training data, we also develop a scalable pipeline to produce synthetic conflict maps using random city model generators and annotated facade images. Extensive experiments demonstrate that FacaDiffy achieves state-of-the-art performance in conflict map completion compared to various inpainting baselines and increases the detection rate by $22\%$ when applying the completed conflict maps for high-definition 3D semantic building reconstruction. The code is be publicly available in the corresponding GitHub repository: https://github.com/ThomasFroech/InpaintingofUnseenFacadeObjects
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。