让视觉语言模型学会看地图,精准导航。
TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation
- 在注意力机制中注入拓扑结构信息,增强空间推理能力。
- 在未见环境中取得51.09%成功率和47.18%SPL,领先9.08点。
- 适合需要全局路径规划的智能体导航任务研究者。
视觉-语言导航(VLN)对大视觉-语言模型(VLM)构成独特挑战,因其架构本质与导航所需的动态、具身化和空间结构特性不匹配。现有方法常将丰富视觉与空间信息转为文本,导致模型需隐式推断复杂视觉拓扑关系或受限于全局动作能力。为此,我们提出TagaVLM(拓扑感知全局动作推理)框架,通过显式注入拓扑结构提升VLM性能。具体地,空间拓扑感知残差注意力(STAR-Att)直接将拓扑边信息融入自注意力机制,实现内在空间推理且保留预训练知识;交错导航提示增强节点级视觉-文本对齐。最终模型具备全局动作推理能力,支持鲁棒路径修正。在R2R基准上,TagaVLM在未见环境中达到51.09%成功率(SR)和47.18%SPL,分别优于前序工作3.39%和9.08点。结果表明,针对小规模开源VLM进行针对性增强,比盲目扩大模型规模更有效于具身空间推理。
原文摘要 · Abstract (English)
Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch: VLMs are primarily pretrained on static, disembodied vision-language tasks, which fundamentally clash with the dynamic, embodied, and spatially-structured nature of navigation. Existing large-model-based methods often resort to converting rich visual and spatial information into text, forcing models to implicitly infer complex visual-topological relationships or limiting their global action capabilities. To bridge this gap, we propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone. To introduce topological edge information, Spatial Topology Aware Residual Attention (STAR-Att) directly integrates it into the VLM's self-attention mechanism, enabling intrinsic spatial reasoning while preserving pretrained knowledge. To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment. Finally, with the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction. On the R2R benchmark, TagaVLM achieves state-of-the-art performance among large-model-based methods, with a Success Rate (SR) of 51.09% and SPL of 47.18 in unseen environments, outperforming prior work by 3.39% in SR and 9.08 in SPL. This demonstrates that, for embodied spatial reasoning, targeted enhancements on smaller open-source VLMs can be more effective than brute-force model scaling. The code can be found on our project page: https://apex-bjut.github.io/Taga-VLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。