提出轻量级导航框架,让AI agent更准地理解语言指令并避障前行。
AgentVLN: Towards Agentic Vision-and-Language Navigation
- 用视觉语言模型做大脑,分离语义推理与感知规划
- 跨空间映射实现3D路径与图像像素对齐,提升定位精度
- 主动追问获取深度信息,适合复杂未结构化环境
视觉-语言导航(VLN)要求具身智能体将复杂的自然语言指令转化为在未知环境中的长程导航。尽管视觉语言模型(VLM)具备强大的2D语义理解能力,现有系统仍受限于空间感知不足、2D-3D表示不一致以及单目尺度模糊等问题。本文提出AgentVLN,一种可部署于边缘计算平台的新型高效具身导航框架。我们将VLN建模为部分可观测半马尔可夫决策过程(POSMDP),引入VLM-as-Brain范式,通过即插即用技能库解耦高层语义推理与感知规划。为解决多层级表示不一致问题,设计跨空间表示映射,将感知层3D拓扑路径点投影至图像平面,生成像素对齐的视觉提示供VLM使用。在此基础上,集成上下文感知的自我修正与主动探索策略,以应对遮挡并抑制长轨迹中的误差累积。为进一步缓解非结构化环境中指令的空间模糊性,提出查询驱动的感知思维链(QD-PCoT)方案,赋予智能体元认知能力,主动寻求几何深度信息。最后,构建AgentVLN-Instruct大规模指令微调数据集,支持基于目标可见性的动态阶段路由。大量实验表明,AgentVLN在长程VLN基准上持续优于先前SOTA方法,为下一代具身导航模型的轻量化部署提供实用范式。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。