构建可扩展的地图到街景空间推理基准,检验模型对地理视角的对齐能力。
m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning
- 设计地图与街景图像对齐任务,判断摄像头朝向。
- 最佳模型仅65.2%准确率,人类平均72.0%,专家达95%。
- 揭示模型在几何对齐与推理一致性上的根本缺陷,适合研究视觉-语言模型空间理解。
视觉-语言模型(VLMs)在多个多模态基准上表现优异,但在需要将抽象俯视地图与第一人称街景视图对齐的空间推理任务中仍显脆弱。我们提出m2sv,一个可扩展的地图到街景空间推理基准,要求模型通过将北向朝上的俯视地图与同一真实路口拍摄的街景图像对齐来推断相机朝向。我们发布m2sv-20k,一个地理多样且控制歧义的基准数据集,以及m2sv-sft-11k,用于监督微调的结构化推理轨迹集合。尽管在现有基准上表现良好,最佳评估的VLM在m2sv上仅达到65.2%准确率,低于人类标注者平均72.0%(专家达95%),且标注者间一致性高(κ最高0.76)。监督微调和强化学习带来稳定提升,但跨基准评估显示迁移能力有限。我们系统分析了地图-街景推理的难度,结合结构信号与人工耗时,并对适配的开源模型进行了详尽故障分析。结果表明,几何对齐、证据聚合与推理一致性仍存在持续差距,推动未来面向多视角的具身空间推理研究。
原文摘要 · Abstract (English)
Vision--language models (VLMs) achieve strong performance on many multimodal benchmarks but remain brittle on spatial reasoning tasks that require aligning abstract overhead representations with egocentric views. We introduce m2sv, a scalable benchmark for map-to-street-view spatial reasoning that asks models to infer camera viewing direction by aligning a north-up overhead map with a Street View image captured at the same real-world intersection. We release m2sv-20k, a geographically diverse benchmark with controlled ambiguity, along with m2sv-sft-11k, a curated set of structured reasoning traces for supervised fine-tuning. Despite strong performance on existing multimodal benchmarks, the best evaluated VLM achieves only 65.2% accuracy on m2sv, below human annotators who reach 72.0% on average (and 95% for an expert) with strong inter-annotator agreement ($κ$ up to 0.76). While supervised fine-tuning and reinforcement learning yield consistent gains, cross-benchmark evaluations reveal limited transfer. Beyond aggregate accuracy, we systematically analyze difficulty in map-to-street-view reasoning using both structural signals and human effort, and conduct an extensive failure analysis of adapted open models. Our findings highlight persistent gaps in geometric alignment, evidence aggregation, and reasoning consistency, motivating future work on grounded spatial reasoning across viewpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。