用文本地图让大模型轻松理解环境，支持千类语义且内存节省百倍。
Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models
- 基于大规模图像模型构建可直接与大模型对接的文本地图
- 定位精度媲美开放词汇地图，内存占用减少2~4个数量级
- 适合需要高效语义理解的机器人导航任务
大型语言模型（LLM）正被用于机器人基于常识推理生成任务计划。为使LLM生成可执行计划，需提供场景上下文，通常依赖地图。近期研究从固定语义类别的显式地图转向基于可查询嵌入的隐式开放词汇地图，能表示任意语义类。但嵌入表示隐式，无法直接提供场景信息，需额外处理才能与LLM集成。为此，我们提出一种显式文本地图，基于大规模图像识别模型，可表示数千种语义类别，且因文本形式天然适配LLM。我们研究了地图中实体的定位方法，实验表明其定位性能与开放词汇地图相当，同时内存使用减少2至4个数量级。真实机器人实验验证了该地图与LLM结合后可成功完成用户任务。
原文摘要 · Abstract (English)
Large Language Models (LLM) have emerged as a tool for robots to generate task plans using common sense reasoning. For the LLM to generate actionable plans, scene context must be provided, often through a map. Recent works have shifted from explicit maps with fixed semantic classes to implicit open vocabulary maps based on queryable embeddings capable of representing any semantic class. However, embeddings cannot directly report the scene context as they are implicit, requiring further processing for LLM integration. To address this, we propose an explicit text-based map that can represent thousands of semantic classes while easily integrating with LLMs due to their text-based nature by building upon large-scale image recognition models. We study how entities in our map can be localized and show through evaluations that our text-based map localizations perform comparably to those from open vocabulary maps while using two to four orders of magnitude less memory. Real-robot experiments demonstrate the grounding of an LLM with the text-based map to solve user tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。