测试大模型对地理坐标的理解能力，发现其在真实地理推理上表现优于几何计算。
GPSBench: Do Large Language Models Understand GPS Coordinates?
- 构建包含57,800样本的GPSBench数据集，涵盖17项地理空间推理任务。
- 模型在国家层级地理知识上表现良好，但城市级定位能力弱，几何计算准确率低。
- 坐标噪声鲁棒性表明模型具备真理解而非记忆，适合导航与机器人应用研究者。
大型语言模型（LLMs）越来越多地应用于导航、机器人和地图等与物理世界交互的任务中，因此稳健的地理空间推理能力至关重要。然而，现有研究尚未充分探索LLMs对地理坐标和现实地理的理解能力。本文提出GPSBench，一个包含57,800个样本、覆盖17项任务的数据集，用于评估LLMs在地理空间推理方面的能力，涵盖几何坐标操作（如距离与方位计算）以及融合坐标与世界知识的推理。聚焦于模型内在能力而非工具使用，我们评估了14个最先进的LLM，发现地理坐标推理仍具挑战性，任务间表现差异显著：模型在真实地理推理上普遍优于几何计算。地理知识呈现层级退化现象，国家层级表现良好，但城市层级定位能力弱；同时，对坐标噪声的鲁棒性表明模型具备真实的坐标理解，而非单纯记忆。此外，我们发现坐标增强可提升下游地理任务性能，而微调虽提升几何计算能力，却导致世界知识下降。数据集与可复现代码已开源：https://github.com/joey234/gpsbench。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。