首个面向城市无人机的视觉语言导航模型,融合多尺度环境信息提升导航精度。
NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation
- 构建多尺度环境表征:全局拓扑图、全景图与细粒度地标
- 在Touchdown和Map2seq上超越强基线,关键在于细粒度地标识别能力
- 发布首个真实城市街景地标数据集NavAgent-Landmark2K,适合户外导航研究者
视觉-语言导航(VLN)是具身智能中的重要研究方向,旨在使智能体通过自然语言指令在复杂视觉环境中导航。现有方法多聚焦于室内地面机器人场景,而应用于室外城市无人机导航时面临两大挑战:一是城市场景物体众多,难以将图像中的细粒度地标与复杂文本描述匹配;二是环境信息包含多模态维度,表示多样性显著增加编码难度。为此,我们提出NavAgent,首个基于大视觉-语言模型的城市无人机具身导航模型。该模型通过融合多尺度环境信息(全局拓扑图、中等尺度全景图、局部细粒度地标)完成导航任务。具体地,采用GLIP构建可识别并语言化细粒度地标的视觉识别器;设计动态生长的场景拓扑图以整合环境信息,并使用图卷积网络编码全局数据。为训练地标识别器,我们构建了首个真实城市街景细粒度地标数据集NavAgent-Landmark2K。在Touchdown与Map2seq数据集上的实验表明,NavAgent优于多个强基线模型。代码与数据集将开源,以推动户外VLN研究发展。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN), as a widely discussed research direction in embodied intelligence, aims to enable embodied agents to navigate in complicated visual environments through natural language commands. Most existing VLN methods focus on indoor ground robot scenarios. However, when applied to UAV VLN in outdoor urban scenes, it faces two significant challenges. First, urban scenes contain numerous objects, which makes it challenging to match fine-grained landmarks in images with complex textual descriptions of these landmarks. Second, overall environmental information encompasses multiple modal dimensions, and the diversity of representations significantly increases the complexity of the encoding process. To address these challenges, we propose NavAgent, the first urban UAV embodied navigation model driven by a large Vision-Language Model. NavAgent undertakes navigation tasks by synthesizing multi-scale environmental information, including topological maps (global), panoramas (medium), and fine-grained landmarks (local). Specifically, we utilize GLIP to build a visual recognizer for landmark capable of identifying and linguisticizing fine-grained landmarks. Subsequently, we develop dynamically growing scene topology map that integrate environmental information and employ Graph Convolutional Networks to encode global environmental data. In addition, to train the visual recognizer for landmark, we develop NavAgent-Landmark2K, the first fine-grained landmark dataset for real urban street scenes. In experiments conducted on the Touchdown and Map2seq datasets, NavAgent outperforms strong baseline models. The code and dataset will be released to the community to facilitate the exploration and development of outdoor VLN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。