实现端到端开放词汇3D语义建图，支持实时更新与回环检测。
Open-Vocabulary Online Semantic Mapping for SLAM
- 用CLIP向量描述3D片段，通过新融合方法从多视角提取特征。
- 计算开销和内存占用低于离线方法，分割性能优于多数现有方案。
- 首次实现神经网络融合CLIP描述符的在线建图，适配多种SLAM系统。
本文提出一种名为OVO的开放词汇在线3D语义建图框架。给定一系列带位姿的RGB-D帧，该方法检测并跟踪3D段，使用CLIP向量进行描述。这些向量通过一种新颖的CLIP融合方法，从观测视角中生成。值得注意的是，OVO的计算开销和内存占用显著低于离线基线方法，同时在分割指标上优于离线与在线方法。除了优越的分割性能，我们还将建图模块集成至两种不同的完整SLAM后端（Gaussian-SLAM与ORB-SLAM2），首次实现基于神经网络融合CLIP描述符的端到端开放词汇在线3D建图，并支持回环闭合。
原文摘要 · Abstract (English)
This paper presents an Open-Vocabulary Online 3D semantic mapping pipeline, that we denote by its acronym OVO. Given a sequence of posed RGB-D frames, we detect and track 3D segments, which we describe using CLIP vectors. These are computed from the viewpoints where they are observed by a novel CLIP merging method. Notably, our OVO has a significantly lower computational and memory footprint than offline baselines, while also showing better segmentation metrics than offline and online ones. Along with superior segmentation performance, we also show experimental results of our mapping contributions integrated with two different full SLAM backbones (Gaussian-SLAM and ORB-SLAM2), being the first ones using a neural network to merge CLIP descriptors and demonstrating end-to-end open-vocabulary online 3D mapping with loop closure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。