arXiv:2506.20671cs.CV2025-06NeurIPS被引 3

提出视觉3D全景场景补全新方法,动态生成实例提案提升精度与效率。

IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals

  • 基于图像上下文动态生成实例提案,测试时也能自适应调整。
  • 在真实数据集上达到顶尖性能,零样本迁移能力显著优于现有方法。
  • 推理速度提升14倍以上,适合实时机器人导航等应用。

语义场景补全(SSC)已成为联合学习场景几何与语义的关键技术,支持移动机器人导航等下游任务。近期扩展至全景场景补全(PSC),通过引入实例级信息增强物体级别感知。尽管PSC已在激光雷达上实现,基于摄像头的方法仍缺乏探索。现有基于Transformer的方法使用固定查询重建场景,虽训练时可更新,但测试时保持静态,难以动态适应观测场景。为此,本文提出IPFormer,首个在训练与测试阶段均采用上下文自适应实例提案的视觉3D全景场景补全方法。具体地,IPFormer从图像上下文自适应初始化全景实例提案,并通过注意力编码解码机制进一步优化,以推理语义实例-体素关系。大量实验表明,该方法在域内表现达领先水平,对域外数据具有优越零样本泛化能力,且运行时间减少超14倍。结果凸显了上下文自适应提案作为视觉3D全景场景补全开创性方案的价值。代码已开源。

原文摘要 · Abstract (English)

Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first method that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Extensive experimental results show that our approach achieves state-of-the-art in-domain performance, exhibits superior zero-shot generalization on out-of-domain data, and achieves a runtime reduction exceeding 14x. These results highlight our introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion. Code available at https://github.com/markus-42/ipformer.

3D补全全景分割视觉感知Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。