视觉语言模型能读懂建筑平面图，助力机器人导航
Vision Language Models Can Parse Floor Plan Maps
- 用视觉语言模型解析平面图，结合图文理解任务路径
- 在九步导航任务中成功率高达96%
- 对开阔区域表现下降，适合研究机器人地图理解的团队
视觉语言模型（VLMs）可同时处理图像与文本，广泛应用于视觉问答、图像描述等任务。本文首次探索将VLM应用于地图解析这一新任务，尤其适用于移动机器人。地图解析不仅需理解标签，还需掌握几何布局与空间连接关系。我们通过提示VLM基于平面图生成复杂室内导航的任务计划，评估其性能。实验结果表明，该模型在包含九个导航动作（如靠近并穿过门）的任务中达到0.96的成功率。除直观发现（小地图和简单任务表现更好）外，还观察到模型在大范围开放区域性能显著下降。我们据此提出有效改进策略，并通过实验验证其可行性。
原文摘要 · Abstract (English)
Vision language models (VLMs) can simultaneously reason about images and texts to tackle many tasks, from visual question answering to image captioning. This paper focuses on map parsing, a novel task that is unexplored within the VLM context and particularly useful to mobile robots. Map parsing requires understanding not only the labels but also the geometric configurations of a map, i.e., what areas are like and how they are connected. To evaluate the performance of VLMs on map parsing, we prompt VLMs with floor plan maps to generate task plans for complex indoor navigation. Our results demonstrate the remarkable capability of VLMs in map parsing, with a success rate of 0.96 in tasks requiring a sequence of nine navigation actions, e.g., approaching and going through doors. Other than intuitive observations, e.g., VLMs do better in smaller maps and simpler navigation tasks, there was a very interesting observation that its performance drops in large open areas. We provide practical suggestions to address such challenges as validated by our experimental results. Webpage: https://sites.google.com/view/vlm-floorplan/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。