将导航语义进展与动作执行分离，提升视觉语言导航的准确性和鲁棒性。
From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation
- 设计双模块框架，分别追踪语义进度和生成局部动作。
- 在R2R-CE上将成功率从48.1%提升至55.3%，路径相似度从43.3%升至48.2%。
- 仅需11.5K动作监督样本，适用于真实室内外环境部署。
视觉-语言导航（VLN）要求智能体根据路线级指令，从视角内视觉观测中执行一系列动作。现有基于视觉语言模型的导航器通常仅通过下一步动作预测进行监督，难以区分进展错误与执行错误。当智能体偏离路线时，纠正动作标签虽可恢复下一步，但无法判断是选择了错误子指令还是正确指令执行失败，导致其持续在错误进展状态下决策。为解决这一歧义，本文提出Route2Step框架，通过显式的步骤级接口将语义进展追踪与动作生成解耦。指令分析模块（$/mathcal{M}_{ ext{IA}}$）基于全局指令与视觉历史预测当前进展状态；动作生成模块（$/mathcal{M}_{ ext{AG}}$）在该状态和近期观测条件下生成局部动作片段。为无需人工时间标注即可监督进展状态，提出E-SPA步骤对齐方法，将子指令与路线级示范中的对应段落关联。该对齐支持对错误进展估计的监督，而直接动作监督仅保留于反复在正确活跃子指令下失败的推演组。在R2R-CE上，Route2Step将成功率从48.1%提升至55.3%，路径相似度从43.3%提升至48.2%，使用190K状态级纠正样本，仅需11.5K直接动作监督状态。真实室内与室外环境实验进一步验证了该方法的实用性。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) requires an agent to follow a route-level instruction by executing its constituent steps from egocentric visual observations. Existing VLM-based navigators typically supervise both capabilities through next-action prediction alone, making progress-tracking errors difficult to distinguish from execution errors. When an agent deviates from the route, a corrective action label may recover the next movement but does not indicate whether the agent selected the wrong sub-instruction or failed to execute the correct one. Consequently, the agent may continue making decisions from an erroneous progress state. To resolve this ambiguity, we propose \textbf{Route2Step}, a framework that decouples semantic progress tracking from action generation through an explicit step-level interface. The Instruction Analysis Module ($\mathcal{M}_{\mathrm{IA}}$) predicts this state from the global instruction and visual history. Conditioned on the predicted state and recent observations, the Action Generation Module ($\mathcal{M}_{\mathrm{AG}}$) generates local action chunks. To supervise the progress state without manual temporal labels, E-SPA, a step-alignment procedure, associates sub-instructions with their corresponding portions of route-level demonstrations. These alignments enable state supervision for incorrect progress estimates, while direct action supervision is reserved for rollout groups that repeatedly fail under the correct active sub-instruction. On R2R-CE, Route2Step improves SR from 48.1\% to 55.3\% and SPL from 43.3\% to 48.2\%, using 190K state-level corrective samples while requiring only 11.5K directly action-supervised states. Experiments in real-world indoor and outdoor environments further demonstrate the practical applicability of Route2Step. The project page is: https://sisyphus-hxy.github.io/Route2Step/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。