单张图像实现跨场景3D平面重建,无需额外训练。
Towards In-the-wild 3D Plane Reconstruction from a Single Image
- 用Transformer模型分离平面法向与偏移量,提升泛化能力。
- 构建超56万标注的多场景平面数据集,覆盖室内外复杂环境。
- 零样本测试下在野外数据上表现显著优于现有方法。
从单张图像进行3D平面重建是3D计算机视觉中的关键但极具挑战性的问题。以往最先进方法通常仅在室内或室外单一数据集上训练,限制了其在多样化测试数据上的泛化能力。本文提出一种名为ZeroPlane的新框架,基于Transformer模型,实现对多样场景和环境的单图像零样本3D平面检测与重建。为支持多领域数据驱动建模,我们构建了一个大规模平面基准数据集,涵盖超过14个数据集和56万张高分辨率、密集平面标注,覆盖多种室内外场景。针对多数据集训练中的几何精度挑战,我们提出解耦平面法向与偏移量的表示,并采用示例引导的分类-回归范式分别学习平面与偏移。此外,采用先进主干网络作为图像编码器,并设计像素-几何增强型平面嵌入模块,进一步促进平面重建效果。在多个零样本评估数据集上的大量实验表明,该方法在重建精度和泛化能力方面均显著优于此前方法,尤其在真实野外数据上表现突出。代码与数据已开源:https://github.com/jcliu0428/ZeroPlane。
原文摘要 · Abstract (English)
3D plane reconstruction from a single image is a crucial yet challenging topic in 3D computer vision. Previous state-of-the-art (SOTA) methods have focused on training their system on a single dataset from either indoor or outdoor domain, limiting their generalizability across diverse testing data. In this work, we introduce a novel framework dubbed ZeroPlane, a Transformer-based model targeting zero-shot 3D plane detection and reconstruction from a single image, over diverse domains and environments. To enable data-driven models across multiple domains, we have curated a large-scale planar benchmark, comprising over 14 datasets and 560,000 high-resolution, dense planar annotations for diverse indoor and outdoor scenes. To address the challenge of achieving desirable planar geometry on multi-dataset training, we propose to disentangle the representation of plane normal and offset, and employ an exemplar-guided, classification-then-regression paradigm to learn plane and offset respectively. Additionally, we employ advanced backbones as image encoder, and present an effective pixel-geometry-enhanced plane embedding module to further facilitate planar reconstruction. Extensive experiments across multiple zero-shot evaluation datasets have demonstrated that our approach significantly outperforms previous methods on both reconstruction accuracy and generalizability, especially over in-the-wild data. Our code and data are available at: https://github.com/jcliu0428/ZeroPlane.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。