arXiv:2607.01079cs.RO2026-07

用视觉语言模型实现室内定位,靠语义地图而非几何计算。

Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization

论文配图:Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization
图 1 · 摘自论文原文
  • 用多模态输入直接回归机器人位姿,跳过文本生成步骤。
  • 在真实数据上达98%定位准确率,0.11米均方误差,推理仅0.62秒。
  • 即使地图不全或物体未见过,仍保持高精度,适合复杂场景应用。

针对无GPS的室内环境,本文将机器人定位重定义为语义推理任务,而非几何估计。基于人类利用物体线索和标注地图定位的启发,研究是否可通过视觉语言模型,结合前视相机图像、极坐标激光雷达扫描和俯视语义地图,推断机器人位姿。采用LoRA微调Qwen2.5-VL-7B,并附加轻量回归头,从最终隐藏状态直接预测连续位姿坐标(x, y, θ),避免文本生成。训练使用包含120,112个样本、527个场景的自定义Gazebo数据集,结合位置与方向联合损失及课程学习。在18,017个样本的分布内测试集上,模型实现98.23%位置准确率、98.00%方向准确率、96.75%完整位姿准确率,平均位置误差0.11米,平均朝向误差5.7度,每样本耗时0.62秒。在七类未见物体上,位置准确率仅下降7.2个百分点至90.99%,表明具备语义空间推理能力而非外观记忆。当地图信息缺失时,微调后仍可恢复至93.72%位置准确率,展现对陈旧或不完整地图的适应性。消融实验显示跨模态互补:仅用相机与地图时,位置准确率达95.06%,比全系统低3.2个百分点;当相机面对墙且无可见物体时,激光雷达仍维持92.33%准确率,而双模态均不可用时降至70.74%。这说明激光雷达在视觉语义失效时成为主信号,可有效应对遮挡或稀疏布局。

原文摘要 · Abstract (English)

We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task rather than a geometric estimation problem. Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-language model, given a front camera image, a polar LiDAR scan, and a top-down semantic grid map, can infer the robot pose. We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates (x, y, theta) directly from the final hidden state, bypassing text generation. Training uses a composite position-and-direction loss with curriculum learning on a custom Gazebo dataset of 120,112 samples and 527 scenes. On the in-distribution test set of 18,017 samples, the model achieves 98.23 percent position accuracy, 98.00 percent direction accuracy, 96.75 percent full pose accuracy, a mean position error of 0.11 m, and a mean orientation error of 5.7 degrees at 0.62 s per sample. Position accuracy drops by only 7.2 percentage points on seven unseen object categories, reaching 90.99 percent, supporting semantic spatial reasoning rather than appearance memorization. With incomplete maps, fine-tuning recovers performance to 93.72 percent position accuracy, showing adaptability to stale or partial map information. Two ablations highlight cross-modal complementarity. Without LiDAR, using only camera and map inputs, position accuracy remains 95.06 percent, only 3.2 percentage points below the full system. However, when the camera sees no visible objects in a wall-facing view, LiDAR sustains 92.33 percent position accuracy, compared with 70.74 percent when neither LiDAR nor visible objects are available. This shows that LiDAR becomes the primary localization signal when camera semantics are unavailable and provides a reliable fallback under occlusion or sparse layouts.

机器人定位视觉语言模型多模态融合语义地图

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。