无需训练，用大模型实现视觉语言导航的精准路径检索与对齐
TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigation
- 分四模块设计，零样本调用LLM和VLM提取地标顺序与路径
- 在R2R-Habitat数据集上优于联合语义地图方法，提升导航精度
- 适用于需要快速部署的视觉语言导航系统
本文提出一种模块化方法解决视觉语言导航（VLN）任务，将问题分解为四个子模块，在零样本设置下使用先进的大语言模型（LLM）和视觉语言模型（VLM）。给定自然语言导航指令后，首先通过LLM提取待访问地标及其顺序；基于已知环境模型，从最后地标检索前k个位置，并利用拓扑图上的最短路径算法生成从起点到该地标的k条路径假设。每条路径由一组全景图序列表示，再通过动态规划计算全景序列与地标名称序列之间的对齐得分，该得分来自VLM匹配结果。最后，计算得分最高的路径假设与参考路径间的nDTW距离以评估路径保真度。实验表明，该方法在复杂的R2R-Habitat指令数据集上表现优于使用联合语义地图的方法（如VLMaps），并详细量化了视觉定位对导航性能的影响。
原文摘要 · Abstract (English)
In this work, we propose a modular approach for the Vision-Language Navigation (VLN) task by decomposing the problem into four sub-modules that use state-of-the-art Large Language Models (LLMs) and Vision-Language Models (VLMs) in a zero-shot setting. Given navigation instruction in natural language, we first prompt LLM to extract the landmarks and the order in which they are visited. Assuming the known model of the environment, we retrieve the top-k locations of the last landmark and generate $k$ path hypotheses from the starting location to the last landmark using the shortest path algorithm on the topological map of the environment. Each path hypothesis is represented by a sequence of panoramas. We then use dynamic programming to compute the alignment score between the sequence of panoramas and the sequence of landmark names, which match scores obtained from VLM. Finally, we compute the nDTW metric between the hypothesis that yields the highest alignment score to evaluate the path fidelity. We demonstrate superior performance compared to other approaches that use joint semantic maps like VLMaps on the complex R2R-Habitat instruction dataset and quantify in detail the effect of visual grounding on navigation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。