arXiv:2511.10376cs.CVcs.RO2025-11中稿 · CVPR被引 16

用多模态3D场景图实现零样本机器人导航,提升泛化能力。

MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation

  • 构建含视觉信息的多模态3D场景图,保留图像关系而非仅用文字
  • 在GOAT-Bench和HM3D-ObjNav上达到当前最佳性能
  • 适合研究零样本导航与开放词汇任务的开发者

具身导航是机器人在真实世界中运行的基础能力。实际部署需具备开放词汇泛化能力和低训练开销,因此更倾向于零样本方法而非特定任务的强化学习训练。然而,现有基于显式3D场景图的零样本方法常将丰富的视觉观测压缩为纯文本关系,导致构建成本高、视觉证据不可逆丢失且词汇受限。为此,我们提出多模态3D场景图(M3DSG),通过动态分配图像替代文本关系边以保留视觉线索。在此基础上,我们设计了MSGNav系统,包含关键子图选择模块以高效推理、自适应词汇更新模块以支持开放词汇,以及闭合回路推理模块以实现精准探索。此外,我们识别出零样本导航中的“最后一公里”问题——即如何确定可行的目标位置及合适最终视角,并提出基于可视性的视角决策模块来明确解决。大量实验表明,MSGNav在具有挑战性的GOAT-Bench和HM3D-ObjNav基准上达到最先进水平。代码将在https://github.com/ylwhxht/MSGNav公开。

原文摘要 · Abstract (English)

Embodied navigation is a fundamental capability for robotic agents operating. Real-world deployment requires open vocabulary generalization and low training overhead, motivating zero-shot methods rather than task-specific RL training. However, existing zero-shot methods that build explicit 3D scene graphs often compress rich visual observations into text-only relations, leading to high construction cost, irreversible loss of visual evidence, and constrained vocabularies. To address these limitations, we introduce the Multi-modal 3D Scene Graph (M3DSG), which preserves visual cues by replacing textual relational edges with dynamically assigned images. Built on M3DSG, we propose MSGNav, a zero-shot navigation system that includes a Key Subgraph Selection module for efficient reasoning, an Adaptive Vocabulary Update module for open vocabulary support, and a Closed-Loop Reasoning module for accurate exploration reasoning. Additionally, we further identify the last mile problem in zero-shot navigation determining the feasible target location with a suitable final viewpoint, and propose a Visibility-based Viewpoint Decision module to explicitly resolve it. Comprehensive experimental results demonstrate that MSGNav achieves state-of-the-art performance on the challenging GOAT-Bench and HM3D-ObjNav benchmark. The code will be publicly available at https://github.com/ylwhxht/MSGNav.

具身导航多模态零样本3D场景图

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。