评测大模型在跨视角地理定位与姿态估计中的表现,发现其定位强但姿态估计弱。
GeoX-Bench: Benchmarking Cross-View Geo-Localization and Pose Estimation Capabilities of Large Multimodal Models
- 构建包含1.08万组全景-卫星图像对的跨视角地理基准测试集
- 25个顶尖大模型在定位任务中表现良好,但姿态估计能力显著下降
- 指令微调可有效提升模型跨视角地理感知能力,适合自动驾驶研究者
大型多模态模型(LMMs)在众多任务中展现出卓越性能,但在跨视角地理定位与姿态估计领域的能力仍不明晰,而这些能力对导航、自动驾驶和室外机器人至关重要。为此,我们提出GeoX-Bench,一个全面的基准测试体系,用于探索和评估LMMs在跨视角地理定位与姿态估计方面的能力。GeoX-Bench包含10,859组覆盖49个国家128座城市的全景-卫星图像对,以及755,976个问答对,其中42,900个用于基准测试,其余用于增强模型能力。基于该基准,我们评估了25个最先进的LMMs在跨视角地理定位与姿态估计任务中的表现,并进一步探究了指令微调的效果。结果表明,尽管当前LMMs在地理定位任务中表现优异,但在更复杂的姿态估计任务中性能显著下降,揭示了未来改进的关键方向;且在GeoX-Bench数据上进行指令微调能显著提升模型的跨视角地理感知能力。代码与数据集已开源。
原文摘要 · Abstract (English)
Large multimodal models (LMMs) have demonstrated remarkable capabilities across a wide range of tasks, however their knowledge and abilities in the cross-view geo-localization and pose estimation domains remain unexplored, despite potential benefits for navigation, autonomous driving, outdoor robotics, \textit{etc}. To bridge this gap, we introduce \textbf{GeoX-Bench}, a comprehensive \underline{Bench}mark designed to explore and evaluate the capabilities of LMMs in \underline{cross}-view \underline{Geo}-localization and pose estimation. Specifically, GeoX-Bench contains 10,859 panoramic-satellite image pairs spanning 128 cities in 49 countries, along with corresponding 755,976 question-answering (QA) pairs. Among these, 42,900 QA pairs are designated for benchmarking, while the remaining are intended to enhance the capabilities of LMMs. Based on GeoX-Bench, we evaluate the capabilities of 25 state-of-the-art LMMs on cross-view geo-localization and pose estimation tasks, and further explore the empowered capabilities of instruction-tuning. Our benchmark demonstrate that while current LMMs achieve impressive performance in geo-localization tasks, their effectiveness declines significantly on the more complex pose estimation tasks, highlighting a critical area for future improvement, and instruction-tuning LMMs on the training data of GeoX-Bench can significantly improve the cross-view geo-sense abilities. The GeoX-Bench is available at \textcolor{magenta}{https://github.com/IntMeGroup/GeoX-Bench}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。