将3D实例分割重构为在线跟踪问题,实现动态环境的实时精准感知。
Online Segment Any 3D Thing as Instance Tracking
- 用稀疏查询实现时序信息传播,结合长短期跟踪保持身份一致性。
- 在ScanNet200上提升2.8 AP,多数据集表现优于ESAM等基线。
- 适合需要实时3D理解的机器人、自动驾驶等场景应用。
在线、实时且细粒度的3D分割是具身智能体感知与理解操作环境的基础能力。现有方法通过预定义的对象查询聚合视觉基础模型(VFMs)输出的语义信息,并将其提升至3D点云,借助查询间交互实现空间信息传播。然而,感知本质上是动态过程,时序理解这一关键维度在现有查询框架中被忽视。为此,我们提出将在线3D分割重新构想为实例跟踪问题(AutoSeg3D)。核心策略是利用对象查询进行时序信息传播:长期实例关联促进特征与身份的一致性,短期实例更新丰富即时观测。由于具身机器人视角变化常导致跨帧物体部分可见,该机制帮助模型在不完整视图下构建整体物体认知。此外,引入空间一致性学习以缓解VFMs固有的碎片化问题,增强长短期时序学习的效果。稀疏对象查询驱动的时序交互与一致性学习不仅提升空间理解,还避免密集点云时序交互带来的计算负担。本方法在ScanNet200上超越ESAM 2.8 AP,且在ScanNet、SceneNN和3RScan上均取得稳定提升。
原文摘要 · Abstract (English)
Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。