构建跨尺度地理定位基准,评估视觉语言模型真实场景下的定位能力。
Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
- 设计双尺度数据集:国家级与街景级定位任务并行测试
- 引入多步推理评分机制,量化模型思考过程中的关键线索贡献
- 发现模型存在区域偏见,且网络搜索不总能提升精度
视觉语言模型虽发展迅速,但在开放世界中基于图像进行地理定位的能力尚未得到全面评估。本文提出 EarthWhere,一个涵盖810张全球分布图像的综合性基准,包含两个互补任务:WhereCountry(500道多选题,答案为国家层级及全景图)和 WhereStreet(310项细粒度街景识别,需多步推理并可选网页搜索)。评估采用最终预测指标:坐标准确率(Acc@k km)和文本定位的层级路径得分。此外,我们提出基于人工验证的关键视觉线索评分与谢林值加权思维分数,以量化每一步推理的贡献。在地球村上对13个前沿视觉语言模型进行评测,结果显示,Gemini-2.5-Pro平均准确率达56.32%,最强开源模型GLM-4.5V为34.71%。研究发现,当视觉线索有限时,网络搜索与推理未必提升性能,且模型表现出显著区域偏见,某些地区表现最高可达其他地区42.7%以上。该结果揭示了模型在消除偏见、实现鲁棒精细定位方面仍面临挑战。基准已开源:https://github.com/UCSC-VLAA/EarthWhere。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have advanced rapidly, yet their capacity for image-grounded geolocation in open-world conditions, a task that is challenging and of demand in real life, has not been comprehensively evaluated. We present EarthWhere, a comprehensive benchmark for VLM image geolocation that evaluates visual recognition, step-by-step reasoning, and evidence use. EarthWhere comprises 810 globally distributed images across two complementary geolocation scales: WhereCountry (i.e., 500 multiple-choice question-answering, with country-level answer and panoramas) and WhereStreet (i.e., 310 fine-grained street-level identification tasks requiring multi-step reasoning with optional web search). For evaluation, we adopt the final-prediction metrics: location accuracies within k km (Acc@k) for coordinates and hierarchical path scores for textual localization. Beyond this, we propose to explicitly score intermediate reasoning chains using human-verified key visual clues and a Shapley-reweighted thinking score that attributes credit to each clue's marginal contribution. We benchmark 13 state-of-the-art VLMs with web searching tools on our EarthWhere and report different types of final answer accuracies as well as the calibrated model thinking scores. Overall, Gemini-2.5-Pro achieves the best average accuracy at 56.32%, while the strongest open-weight model, GLM-4.5V, reaches 34.71%. We reveal that web search and reasoning do not guarantee improved performance when visual clues are limited, and models exhibit regional biases, achieving up to 42.7% higher scores in certain areas than others. These findings highlight not only the promise but also the persistent challenges of models to mitigate bias and achieve robust, fine-grained localization. We open-source our benchmark at https://github.com/UCSC-VLAA/EarthWhere.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。