arXiv:2606.00095cs.CVcs.AI2026-06被引 1

构建分层语义几何地图,让AI导航更懂空间关系

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

论文配图:Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
图 1 · 摘自论文原文
  • 用三层地图融合语义与几何信息,连接视觉语言模型与物理空间
  • 零样本测试下在R2R-CE和RxR-CE上达顶尖性能,超越部分有监督方法
  • 适合做智能机器人导航、具身AI的开发者与研究者参考

视觉语言导航(VLN)使智能体能根据语言指令在未见过的环境中到达目标位置。尽管视觉语言模型(VLMs)在理解和2D视觉方面表现优异,但仍存在显著的语义-几何鸿沟:难以进行3D空间推理,无法捕捉动作与空间变化间的因果关系,导致零样本场景下导航不可靠。为此,我们提出分层语义-几何地图(HSGM),将3D几何信息转化为适配VLMs的结构化表示,有效连接模型与物理世界。HSGM为多通道俯视图,分为三级:(1) 几何级记录可通行区域与障碍物,(2) 语义级表示物体及其关系,(3) 决策级支持高层任务推理与目标选择。导航时,VLM作为高层语义规划器,基于编码的空间布局选择几何上可行的航点;低层无碰撞路径由经典路径规划算法执行,实现语义推理与动作执行的完全解耦。复杂指令被分解为子任务,缓解长序列导航中的进展遗忘或幻觉问题。在R2R-CE和RxR-CE基准上的大量实验表明,该零样本框架达到当前最优性能,甚至超过多个有监督方法。代码已开源。

原文摘要 · Abstract (English)

Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions. Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains: while VLMs excel at language and 2D visual understanding, they struggle with 3D spatial reasoning and fail to capture the causal dynamics between actions and spatial transitions, resulting in unreliable navigation, particularly in zero-shot settings. To bridge this gap, we propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible with VLMs, effectively linking them to the physical world. Specifically, HSGM is represented as a multi-channel top-down map organized into three levels: (1) geometric level that records navigable regions and obstacles, (2) semantic level that represents objects and their relations, and (3) decision level that supports high-level task reasoning and goal selection. During navigation, the VLM acts as a high-level semantic planner, interpreting the spatial layout encoded in the HSGM to select geometrically valid waypoints, while low-level, collision-free movements between waypoints are executed by a classical path-planning algorithm, fully decoupling semantic reasoning from action execution. Additionally, complex instructions are decomposed into subtasks to alleviate the problem of progress forgetting or hallucinating in long-horizon navigation. Extensive experiments on R2R-CE and RxR-CE benchmarks demonstrate that our zero-shot framework achieves state-of-the-art performance and even outperforms several supervised methods. Code is available at https://github.com/Teacher-Tom/HSGM_public.

视觉导航语义地图具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。