用自然语言指令生成可执行的全球导航代价图,支持未知实体和复杂规则。
OVerSeeC: Open-Vocabulary Costmap Generation from Satellite Images and Natural Language
- 分三步解析:理解指令、定位目标、合成代价函数
- 在多地区测试中路径与人工绘制轨迹一致,适应新实体和组合规则
- 适合需要灵活规划的长距离自动驾驶任务
航拍图像为自主导航提供全局上下文,支持车载传感器无法覆盖的远距离路径规划。本文解决在测试时通过自然语言表达实体与任务特定通行规则的情况下,直接从卫星图像生成全局代价图的问题。该场景挑战在于任务需求多变、地形实体部署时未知,且用户提示常包含复合通行逻辑。现有依赖固定本体和静态代价映射的方法难以应对这种灵活性。尽管基础模型在语言理解和开放词汇感知方面表现优异,但尚无单一模型能同时解析细微任务指令、在大规模图像中定位任意实体,并将其合成供规划器使用的可执行代价函数。为此,我们提出 OVerSeeC,一种零样本模块化框架,将问题分解为:(i) 大语言模型(LLM)提取实体及优先级排序;(ii) 开放词汇分割管线从高分辨率影像中识别这些实体;(iii) LLM 结合用户自然语言偏好和掩码,合成可执行的代价图代码。实证表明,OVerSeeC 能处理新出现的实体,遵循优先级和复合偏好,并在多个区域生成与人工绘制轨迹一致的路径,表现出对分布偏移的鲁棒性。这证明,基础模型的模块化组合可实现开放词汇、偏好对齐的代价图生成,支持可扩展、任务自适应的全局规划。
原文摘要 · Abstract (English)
Aerial imagery provides essential global context for autonomous navigation, enabling route planning at scales inaccessible to onboard sensing. We address the problem of generating global costmaps for long-range planning directly from satellite imagery when entities and mission-specific traversal rules are expressed in natural language at test time. This setting is challenging since mission requirements vary, terrain entities may be unknown at deployment, and user prompts often encode compositional traversal logic. Existing approaches relying on fixed ontologies and static cost mappings cannot accommodate such flexibility. While foundation models excel at language interpretation and open-vocabulary perception, no single model can simultaneously parse nuanced mission directives, locate arbitrary entities in large-scale imagery, and synthesize them into an executable cost function for planners. We therefore propose OVerSeeC, a zero-shot modular framework that decomposes the problem into Interpret-Locate-Synthesize: (i) an LLM extracts entities and ranked preferences, (ii) an open-vocabulary segmentation pipeline identifies these entities from high-resolution imagery, and (iii) the LLM uses the user's natural language preferences and masks to synthesize executable costmap code. Empirically, OVerSeeC handles novel entities, respects ranked and compositional preferences, and produces routes consistent with human-drawn trajectories across diverse regions, demonstrating robustness to distribution shifts. This shows that modular composition of foundation models enables open-vocabulary, preference-aligned costmap generation for scalable, mission-adaptive global planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。