用视觉语言模型直接一步完成导航，零样本即能用。
End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
- 直接用VLM一次性选择动作，无需分步处理感知、规划与控制。
- 零样本下在多个任务上表现优于基线提示方法，无需微调或导航数据训练。
- 适合快速部署到新导航场景，尤其适合追求通用性与低开发成本的研究者。
我们提出VLMnav，一个将视觉语言模型（VLM）转化为端到端导航策略的具身框架。与以往工作不同，本方法不依赖感知、规划和控制的分离，而是直接使用VLM一步选择动作。令人惊讶的是，VLM可在未经过任何微调或导航数据训练的情况下，实现零样本端到端导航。这使得该方法具有开放性和广泛适用性，可推广至任意下游导航任务。我们进行了大量实验，评估该方法在多种基准上的表现，并开展设计分析以理解关键决策的影响。项目可视化示例及代码详见 https://jirl-upenn.github.io/VLMnav/
原文摘要 · Abstract (English)
We present VLMnav, an embodied framework to transform a Vision-Language Model (VLM) into an end-to-end navigation policy. In contrast to prior work, we do not rely on a separation between perception, planning, and control; instead, we use a VLM to directly select actions in one step. Surprisingly, we find that a VLM can be used as an end-to-end policy zero-shot, i.e., without any fine-tuning or exposure to navigation data. This makes our approach open-ended and generalizable to any downstream navigation task. We run an extensive study to evaluate the performance of our approach in comparison to baseline prompting methods. In addition, we perform a design analysis to understand the most impactful design decisions. Visual examples and code for our project can be found at https://jirl-upenn.github.io/VLMnav/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。