用跨注意力建模3D实例与场景的关联，实现端到端开放词汇全景重建。
Leverage Cross-Attention for End-to-End Open-Vocabulary Panoptic Reconstruction
- 用可学习的3D高斯作为实例查询，融合空间先验保持物体邻近关系。
- 动态调整实例数量，2D/3D分割和重建在仿真与真实数据上表现优秀。
- 适合做机器人仿真与具身智能的全景理解系统开发者参考。
开放词汇全景重建提供全面的场景理解，推动具身机器人与逼真模拟的发展。本文提出PanopticRecon++，一种端到端方法，通过新颖的跨注意力视角将全景重建建模为3D实例（作为查询）与场景3D嵌入场（作为键）之间的关系。该方法利用注意力图建模二者关联，不同于以往分离优化查询与键或忽略空间邻近性的做法，引入可学习的3D高斯作为实例查询，注入3D空间先验以保持邻近性同时维持端到端可优化性。此外，该查询形式支持通过最优线性分配对齐帧间2D开放词汇实例ID，利用查询生成的实例掩码完成对齐。我们还设计了新型全景头，融合基于查询的实例分割概率与语义概率，由全景损失监督，确保语义-实例分割一致性。训练中实例查询标记数动态适配物体数量。PanopticRecon++在仿真与真实世界数据集上均表现出色，2D/3D分割与重建性能具有竞争力，并成功应用于机器人模拟场景。项目主页：https://yuxuan1206.github.io/panopticrecon_pp/
原文摘要 · Abstract (English)
Open-vocabulary panoptic reconstruction offers comprehensive scene understanding, enabling advances in embodied robotics and photorealistic simulation. In this paper, we propose PanopticRecon++, an end-to-end method that formulates panoptic reconstruction through a novel cross-attention perspective. This perspective models the relationship between 3D instances (as queries) and the scene's 3D embedding field (as keys) through their attention map. Unlike existing methods that separate the optimization of queries and keys or overlook spatial proximity, PanopticRecon++ introduces learnable 3D Gaussians as instance queries. This formulation injects 3D spatial priors to preserve proximity while maintaining end-to-end optimizability. Moreover, this query formulation facilitates the alignment of 2D open-vocabulary instance IDs across frames by leveraging optimal linear assignment with instance masks rendered from the queries. Additionally, we ensure semantic-instance segmentation consistency by fusing query-based instance segmentation probabilities with semantic probabilities in a novel panoptic head supervised by a panoptic loss. During training, the number of instance query tokens dynamically adapts to match the number of objects. PanopticRecon++ shows competitive performance in terms of 3D and 2D segmentation and reconstruction performance on both simulation and real-world datasets, and demonstrates a user case as a robot simulator. Our project website is at: https://yuxuan1206.github.io/panopticrecon_pp/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。