用语言预测未来状态，让导航模型更省资源、更高效。
LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory
- 用语言标签生成动作的未来状态，避免依赖复杂地图和历史帧
- 在R2R-CE未见场景上达到49.7%成功率，训练数据更少
- 事件驱动的滚动记忆机制，只存关键信息，内存开销更低
视觉语言导航（VLN）要求智能体在未知环境中根据自然语言指令行动。当前方法主要依赖多模态大模型，采用逐步动作预测范式，仅监督专家动作，需大量数据训练。同时依赖认知地图、历史帧或外部3D工具维护状态，导致计算与内存开销高。为实现资源高效的VLN，我们提出LookStep：一个统一的端到端框架，结合语言中心的未来状态建模与事件驱动的滚动记忆机制。该框架利用语言标签生成每个候选动作的粗粒度导航进展与未来状态，并自主决定是否将观察结果写入有界滚动记忆（基于语义角色）。实验验证表明，在VLN-CE任务中，LookStep在相同训练条件下优于现有方法，在R2R-CE Val-Unseen上取得49.7%的成功率，同时具备更高的内存效率和更低的数据需求。代码与模型已开源。
原文摘要 · Abstract (English)
Vision-Language Navigation (VLN) requires an embodied agent to follow natural-language instructions in unseen environments. Recent progress has been largely driven by Multimodal Large Language Models (MLLMs). Existing methods follow a next-step action prediction paradigm, supervising only the expert action, which requires a high quantity of data for training. They also rely on cognitive maps, accumulated historical frames, or external 3D tools to maintain states, leading to high computational and memory overhead. To realize resource efficiency VLN, we propose LookStep, a unified end-to-end framework that combines Language Centric Future State Modeling and Event Driven Rolling Memory that uses language labels to generate coarse-grained navigation progress and future states for each candidate action, while autonomously deciding whether to write each observation into a bounded rolling memory with a semantic role. We validate LookStep empirically. On VLN-CE tasks, LookStep outperforms existing methods under the same training settings, achieving a 49.7\% success rate on R2R-CE Val-Unseen with better memory efficiency and less data usage. Code and model is available at https://github.com/kunyang-YU/LookStep.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。