让大模型直接操控视觉导航,实现零样本高效路径规划。
AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

- 将导航动作设计为可调用工具,直接选目标像素生成移动指令。
- 引入深度查询与紧凑地图记忆,显著提升路径规划精度。
- 适合研究零样本视觉语言导航与智能体交互的学者使用。
连续环境中的零样本视觉-语言导航(VLN-CE)借助大型视觉语言模型(VLM)已变得可行。然而,现有方法通常依赖学习到的航点预测器来生成可执行动作,严重限制了动作空间并难以有效利用深度信息。此外,记忆管理常通过积累冗长的文本或视觉历史,或检索跨回合经验,削弱了零样本设定的纯粹性。本文重新思考零样本VLN-CE为大模型与环境之间的智能体接口,提出AgenticNav——一种轻量级导航框架,将动作、深度和记忆作为可调用工具暴露。动作工具允许大模型直接在RGB观测中选择目标像素,转换为可执行运动;深度通过按需调用的像素深度工具提供,仅在必要时获取精确距离;记忆则由一张紧凑轨迹图和一个可选回溯工具构成,支持选择性回顾过去视觉观察,避免提示上下文过载。在R2R-CE基准上,给定相同VLM主干,AgenticNav实现了零样本方法的新最佳性能。真实世界验证进一步凸显其优于以往方法的零样本泛化能力。消融实验表明,我们的动作工具设计优于传统航点预测器,深度工具与智能体记忆也显著提升导航表现。
原文摘要 · Abstract (English)
Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). However, existing methods typically rely on learned waypoint predictors to propose navigable actions. This severely limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav provides a compact map image summarizing the historical trajectory, paired with a recall tool that allows the VLM to selectively revisit past visual observations without overwhelming the prompt context. On the R2R-CE benchmark, AgenticNav establishes new state-of-the-art (SOTA) performance among zero-shot methods given the same VLM backbone. Real-world validation further highlights its zero-shot generalization compared to prior methods. Ablations show that our action tool design outperforms traditional waypoint predictors, and that depth tool and agentic memory further contribute to navigation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。