让智能体提前预测未来视觉状态,提升长距离导航能力。
Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation

- 用未来查询标记引导策略学习未来视觉状态
- 在长路径任务上成功率提升,最优路径率提高12.3%
- 仅需少量额外参数,可直接部署于真实场景
端到端视觉语言导航(VLN)使用因果视觉语言模型将指令和自我视角观测直接映射为动作,但标准行为克隆仅监督下一步动作,未显式鼓励策略状态预测未来视觉结果,限制了长时程决策。特权输入诊断表明,若能访问专家轨迹的未来图像,导航性能显著提升,说明未来观测包含丰富可行动线索,但部署时不可用。受此启发,我们提出可部署的未来状态条件化视觉语言导航(FSC-VLN),在因果策略中引入未来查询标记,并通过仅训练阶段的未来状态监督,将未来观测信息提炼至策略状态。具体而言,训练时将未来查询表示对齐至Δ步前的冻结视觉嵌入,推理时仅需过去与当前观测。该设计保持基线推理模式,仅增加两个可学习前缀标记,开销极低。在R2R val-unseen数据集上,FSC-VLN在两种训练数据设置下均优于StreamVLN类基线,尤其在长路径任务中增益更大;消融实验进一步验证了分离未来与动作查询的双查询设计的有效性。
原文摘要 · Abstract (English)
End-to-end vision-language navigation (VLN) with causal vision-language models maps instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly encourage the policy state to be predictive of future visual outcomes, limiting long-horizon decision making. A privileged-input diagnostic shows that access to an expert-trajectory future image can substantially improve navigation, indicating that future observations contain rich, actionable cues, though such inputs are unavailable at deployment. Motivated by this signal, we propose Future-State-Conditioned VLN (FSC-VLN), a deployable model that augments a causal policy with a future-query token and uses training-only future-state supervision to distill information from future observations into the policy state. Concretely, during training we align the future-query representation to a frozen visual embedding $Δ$ steps ahead, while inference requires only past and current observations. This design preserves the baseline inference pattern and adds only two learned prefix tokens, implying minimal overhead. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design that separates future and action queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。