提出分阶段导航框架,让智能体更准确理解语言指令。
P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigation

- 将导航分为全景选向与俯视定位两步,分离方向判断与细节定位
- 在R2R-CE上比顶尖零样本方法提升146.6%成功率
- 支持长距离导航并可自我修正,适合复杂环境应用
视觉-语言导航(VLN)要求智能体在未见过的环境中将自然语言指令转化为可执行的导航动作。现有零样本方法通常依赖额外的路径点预测模块,常导致高层方向推理与低层局部定位纠缠,引发错误且不稳定的决策。本文提出P2DNav,一种分层零样本视觉-语言导航框架。其包含三个核心组件:全景到俯视(P2D)、滑动窗口对话记忆(SDM)和反射重定向机制(RRM)。P2D显式将导航决策分解为两个阶段:先从360°全景图中选择与指令相关的方向,再在该方向的俯视图像中预测像素级目标点。SDM将导航历史组织为多轮对话上下文,并在滑动窗口内维护近期视觉观测,支持长程导航。RRM通过评估俯视观测中局部定位的可靠性,必要时返回全景方向选择进行反射重定向。在R2R-CE基准测试中,P2DNav表现优异,相比最先进的基于路径点和无路径点的零样本方法,成功率达到提升146.6%和58.9%,验证了P2D、SDM与RRM的有效性。代码将公开。
原文摘要 · Abstract (English)
Vision-and-language navigation (VLN) requires an embodied agent to ground natural-language instructions into executable navigation actions in unseen environments. Existing zero-shot methods typically rely on additional waypoint prediction modules, which often entangle high-level directional reasoning with fine-grained local grounding, leading to error-prone and unstable decisions. In this paper, we propose P2DNav, a hierarchical framework for zero-shot vision-and-language navigation. P2DNav consists of three core components: Panorama-to-Downview (P2D), Sliding-Window Dialogue Memory (SDM), and Reflective Reorientation Mechanism (RRM). P2D explicitly decomposes navigation decision-making into two stages: panoramic direction selection and downview local grounding. It first selects the instruction-relevant direction from a 360° panorama, and then predicts a pixel-level target point from the downview RGB observation in that direction. In addition, SDM organizes navigation history as a multi-turn dialogue context and maintains recent visual observations within a sliding window to support long-horizon navigation. RRM further enables reflective reorientation by assessing the reliability of local grounding based on the downview observation and returning to panoramic direction selection when necessary. Experiments on the R2R-CE benchmark show that P2DNav achieves strong performance among zero-shot methods. In particular, compared with the state-of-the-art (SOTA) zero-shot waypoint-based and waypoint-free methods, P2DNav achieves SR gains of 146.6% and 58.9%, respectively, demonstrating the effectiveness of P2D, SDM, and RRM for zero-shot VLN. Code will be released for public use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。