用视觉提示让多模态大模型零样本判断越野路面是否可通行
Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs
- 用SAM2分割环境，再用VLM结合数值标签推理可行驶区域
- 在高分辨率数据集上超越现有训练模型，实现端到端越野导航
- 无需针对地形单独训练模型，适合快速部署于复杂野外场景
传统越野自主系统依赖多个独立模型分别完成地形分类、高程估计和打滑/坡度评估。这类方法需为每个任务单独训练、使用特定数据集并进行微调。本文提出一种零样本方法，利用SAM2进行环境分割，并结合视觉语言模型（VLM）推理可行驶区域。将原始图像与标注了数值标签的分割图像一同输入VLM，由其判断哪些以数字标记的区域可通行。该框架与规划控制模块结合，无需显式训练地形专用模型，仅依赖VLM的内在推理能力。实验表明，该方法在高分辨率分割数据集上超越现有可训练模型，并在Isaac Sim越野环境中实现了完整栈导航。
原文摘要 · Abstract (English)
Traditional approaches to off-road autonomy rely on separate models for terrain classification, height estimation, and quantifying slip or slope conditions. Utilizing several models requires training each component separately, having task specific datasets, and fine-tuning. In this work, we present a zero-shot approach leveraging SAM2 for environment segmentation and a vision-language model (VLM) to reason about drivable areas. Our approach involves passing to the VLM both the original image and the segmented image annotated with numeric labels for each mask. The VLM is then prompted to identify which regions, represented by these numeric labels, are drivable. Combined with planning and control modules, this unified framework eliminates the need for explicit terrain-specific models and relies instead on the inherent reasoning capabilities of the VLM. Our approach surpasses state-of-the-art trainable models on high resolution segmentation datasets and enables full stack navigation in our Isaac Sim offroad environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。