arXiv:2505.10928cs.LG2025-05被引 2

构建首个面向兴趣点的时空问答数据集,评测大模型的时空推理能力。

A Dataset for Spatiotemporal-Sensitive POI Question Answering

  • 基于轨迹与地理数据融合构建时空敏感问答对
  • 顶尖模型在简单任务上准确率仅41%,远低于人类56%
  • 适合研究时空推理、多模态理解的学者使用

时空关系在数据科学中至关重要,许多预测与推理任务需同时分析空间与时间维度。例如,在陌生城市导航需规划包含地点与时间顺序的文化体验行程。然而现有问答数据集缺乏足够的时空敏感问题,难以有效评估模型的时空推理能力。为此,我们提出POI-QA,一个以兴趣点(POI)为中心的新型时空敏感问答数据集,通过三个关键步骤构建:从GAIA开源车辆轨迹数据中挖掘并对齐高精度地理POI数据,人工严格验证存在噪声的时空事实,生成反映人类可理解时空推理任务的中英文双语问答对。该数据集挑战模型对复杂时空依赖关系的解析能力。对主流多语言大模型(如Qwen2.5-7B、Llama3.1-8B)的评估显示显著局限:即使表现最佳的模型(经RAG+LoRA微调的Qwen2.5-7B)在最简单任务上也仅达HR@10=0.41,远低于人类水平的0.56。这凸显了大模型在一致时空推理方面的持续缺陷,同时表明POI-QA是推动时空动态敏感算法发展的可靠基准。数据集已公开于https://www.kaggle.com/ds/7394666。

原文摘要 · Abstract (English)

Spatiotemporal relationships are critical in data science, as many prediction and reasoning tasks require analysis across both spatial and temporal dimensions--for instance, navigating an unfamiliar city involves planning itineraries that sequence locations and timing cultural experiences. However, existing Question-Answering (QA) datasets lack sufficient spatiotemporal-sensitive questions, making them inadequate benchmarks for evaluating models' spatiotemporal reasoning capabilities. To address this gap, we introduce POI-QA, a novel spatiotemporal-sensitive QA dataset centered on Point of Interest (POI), constructed through three key steps: mining and aligning open-source vehicle trajectory data from GAIA with high-precision geographic POI data, rigorous manual validation of noisy spatiotemporal facts, and generating bilingual (Chinese/English) QA pairs that reflect human-understandable spatiotemporal reasoning tasks. Our dataset challenges models to parse complex spatiotemporal dependencies, and evaluations of state-of-the-art multilingual LLMs (e.g., Qwen2.5-7B, Llama3.1-8B) reveal stark limitations: even the top-performing model (Qwen2.5-7B fine-tuned with RAG+LoRA) achieves a top 10 Hit Ratio (HR@10) of only 0.41 on the easiest task, far below human performance at 0.56. This underscores persistent weaknesses in LLMs' ability to perform consistent spatiotemporal reasoning, while highlighting POI-QA as a robust benchmark to advance algorithms sensitive to spatiotemporal dynamics. The dataset is publicly available at https://www.kaggle.com/ds/7394666.

时空推理问答系统数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。