arXiv:2410.04529cs.CV2024-10被引 1

用视觉先验提升3D场景理解,实现更准的全景分割。

In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding

  • 将全景理解建模为2D语义与实例识别的线性分配问题。
  • 在合成与真实场景中,3D分割准确率显著优于现有方法。
  • 适合需要高精度3D场景理解的自动驾驶与机器人应用。

精确的3D场景表征与全景理解对虚拟现实、机器人及自动驾驶等应用至关重要。然而,现有方法仍面临2D到3D映射不准、边界模糊与尺度变化复杂、全景伪标签噪声等问题。本文提出一种基于感知先验的3D场景表征与全景理解新方法,将神经辐射场中的全景理解重构为涉及2D语义与实例识别的线性分配问题。利用预训练2D全景分割模型的感知信息作为先验指导,同步优化外观、几何与全景理解的学习过程。通过在重参数化域蒸馏框架中扩展尺度编码级联网格,构建隐式场景表征与理解模型,有效处理复杂场景属性,在室内与室外场景中均实现3D一致性表征与分割结果。在合成与真实世界场景下的实验与消融研究验证了该方法在提升3D场景表征与全景分割准确性方面的有效性。

原文摘要 · Abstract (English)

Accurate 3D scene representation and panoptic understanding are essential for applications such as virtual reality, robotics, and autonomous driving. However, challenges persist with existing methods, including precise 2D-to-3D mapping, handling complex scene characteristics like boundary ambiguity and varying scales, and mitigating noise in panoptic pseudo-labels. This paper introduces a novel perceptual-prior-guided 3D scene representation and panoptic understanding method, which reformulates panoptic understanding within neural radiance fields as a linear assignment problem involving 2D semantics and instance recognition. Perceptual information from pre-trained 2D panoptic segmentation models is incorporated as prior guidance, thereby synchronizing the learning processes of appearance, geometry, and panoptic understanding within neural radiance fields. An implicit scene representation and understanding model is developed to enhance generalization across indoor and outdoor scenes by extending the scale-encoded cascaded grids within a reparameterized domain distillation framework. This model effectively manages complex scene attributes and generates 3D-consistent scene representations and panoptic understanding outcomes for various scenes. Experiments and ablation studies under challenging conditions, including synthetic and real-world scenes, demonstrate the proposed method's effectiveness in enhancing 3D scene representation and panoptic segmentation accuracy.

3D理解全景分割神经辐射场

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。