让多智能体零样本导航开放词汇目标,靠大模型理解自由语言指令。
GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System
- 用视觉语言模型直接参与决策,实现零样本开放词汇导航。
- 2个智能体在360个任务中达成55.8%子任务成功率,18.3%SPL。
- 适合需要灵活应对新目标的多智能体系统研发者。
传统物体目标导航局限于地面机器人与固定类别词汇表。现有多智能体方法依赖预计算的概率图,无法泛化至测试时的新目标。我们提出GoalVLM,一种协作式多智能体框架,支持零样本、开放词汇物体导航。GoalVLM将视觉语言模型(VLM)直接嵌入决策循环,结合SAM3实现文本提示检测与分割,以及SpaceOM进行空间推理,使智能体能解析自由格式语言指令,并通过零样本语义先验评分前沿区域,无需重新训练。每个智能体通过深度投影体素点云构建鸟瞰语义地图,而目标投影器通过校准深度将检测结果反投影至地图以实现可靠定位。约束引导推理层通过结构化提示链(场景描述、房间类型分类、感知门控、多前沿排序)评估前沿,注入常识先验。我们在GOAT-Bench val_unseen(360个多子任务实验,1032个顺序目标子任务,HM3D场景)上评估,每轮需导航至5-7个开放词汇目标。使用N=2智能体的GoalVLM达成55.8%子任务成功率和18.3%SPL,性能媲美最先进方法,且无需任务特定训练。消融实验验证了VLM引导的前沿推理和深度投影目标定位的有效性。
原文摘要 · Abstract (English)
Object-goal navigation has traditionally been limited to ground robots with closed-set object vocabularies. Existing multi-agent approaches depend on precomputed probabilistic graphs tied to fixed category sets, precluding generalization to novel goals at test time. We present GoalVLM, a cooperative multi-agent framework for zero-shot, open-vocabulary object navigation. GoalVLM integrates a Vision-Language Model (VLM) directly into the decision loop, SAM3 for text-prompted detection and segmentation, and SpaceOM for spatial reasoning, enabling agents to interpret free-form language goals and score frontiers via zero-shot semantic priors without retraining. Each agent builds a BEV semantic map from depth-projected voxel splatting, while a Goal Projector back-projects detections through calibrated depth into the map for reliable goal localization. A constraint-guided reasoning layer evaluates frontiers through a structured prompt chain (scene captioning, room-type classification, perception gating, multi-frontier ranking), injecting commonsense priors into exploration. We evaluate GoalVLM on GOAT-Bench val_unseen (360 multi-subtask episodes, 1032 sequential object-goal subtasks, HM3D scenes), where each episode requires navigating to a chain of 5-7 open-vocabulary targets. GoalVLM with N=2 agents achieves 55.8% subtask SR and 18.3% SPL, competitive with state-of-the-art methods while requiring no task-specific training. Ablation studies confirm the contributions of VLM-guided frontier reasoning and depth-projected goal localization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。