构建动态地理探索评测基准,检验大模型在充电场景下的多目标规划能力。
EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration
- 基于实时位置和充电动机+活动偏好双目标设计评测任务
- 模型能用工具完成子任务但缺乏长程空间探索能力
- 发现模型可总结历史轨迹提升探索效率,适合地理智能研究者
尽管大语言模型具备强大的推理能力,其在动态地理环境中进行目的驱动探索的潜力仍待深入研究。现有地理问答基准多聚焦静态检索,无法反映真实世界规划中动态用户位置与复合约束的复杂性。为此,我们提出EVGeoQA,一个基于电动汽车充电场景的新颖评测基准,具有位置锚定和双目标特性:每个查询明确绑定用户实时坐标,并同时包含充电需求与共处活动偏好。为系统评估模型在此类复杂场景中的表现,我们进一步提出GeoRover——一种基于工具增强代理架构的通用评估框架。实验表明,尽管模型能利用工具解决子任务,但在长距离空间探索上表现不佳。值得注意的是,我们观察到一种涌现能力:模型可通过总结历史探索轨迹来提升探索效率。这些发现确立了EVGeoQA作为未来地理智能研究的挑战性测试平台。数据集与提示模板已开源于https://github.com/kg-bnu/EVGeoQA。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial environments remains under-investigated. Existing Geo-Spatial Question Answering (GSQA) benchmarks predominantly focus on static retrieval, failing to capture the complexity of real-world planning that involves dynamic user locations and compound constraints. To bridge this gap, we introduce EVGeoQA, a novel benchmark built upon Electric Vehicle (EV) charging scenarios that features a distinct location-anchored and dual-objective design. Specifically, each query in EVGeoQA is explicitly bound to a user's real-time coordinate and integrates the dual objectives of a charging necessity and a co-located activity preference. To systematically assess models in such complex settings, we further propose GeoRover, a general evaluation framework based on a tool-augmented agent architecture to evaluate the LLMs' capacity for dynamic, multi-objective exploration. Our experiments reveal that while LLMs successfully utilize tools to address sub-tasks, they struggle with long-range spatial exploration. Notably, we observe an emergent capability: LLMs can summarize historical exploration trajectories to enhance exploration efficiency. These findings establish EVGeoQA as a challenging testbed for future geo-spatial intelligence. The dataset and prompts are available at https://github.com/kg-bnu/EVGeoQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。