构建驾驶场景空间推理基准，评估并提升视觉语言模型的几何理解能力
SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models
- 基于nuScenes数据集构建41,080个训练样本的视觉问答集，覆盖六类空间关系
- 提出强化学习对齐方法，使模型在真实驾驶场景中空间推理得分达40.80，超越GPT-4o和Gemini
- 首次证明强化学习可系统性提升视觉语言模型的空间推理能力，适合自动驾驶研究者使用
准确理解户外环境中的空间关系——包括几何、物体姿态及对象间关系——是自主驾驶中地图构建、运动预测与高层规划等任务的基础。我们提出SURDS，一个大规模基准，用于系统评估视觉语言模型（VLMs）的空间推理能力。该基准基于nuScenes数据集，包含41,080个训练样本和9,250个评估样本，涵盖六类空间关系：方向、深度估计、像素级定位、成对距离、横向排序以及前后关系。我们测试了GPT、Gemini、Qwen等主流通用VLM，发现其在细粒度空间理解上存在明显局限。为解决此问题，我们突破静态评估，探索通过对齐技术提升性能。提出一种基于强化学习的对齐方案，利用空间感知奖励信号——同时捕捉感知精度（位置）与逻辑一致性（推理）。进一步引入最终答案正确性与输出格式奖励，引导精细策略优化。经GRPO对齐的模型在SURDS上获得40.80分，显著优于GPT-4o（13.30）与Gemini-2.0-flash（35.71）。据我们所知，这是首个证明强化学习对齐能显著且一致提升VLM在真实驾驶场景中空间推理能力的研究。我们已开源SURDS基准、评估工具包及GRPO对齐代码：https://github.com/XiandaGuo/Drive-MLLM。
原文摘要 · Abstract (English)
Accurate spatial reasoning in outdoor environments - covering geometry, object pose, and inter-object relationships - is fundamental to downstream tasks such as mapping, motion forecasting, and high-level planning in autonomous driving. We introduce SURDS, a large-scale benchmark designed to systematically evaluate the spatial reasoning capabilities of vision language models (VLMs). Built on the nuScenes dataset, SURDS comprises 41,080 vision-question-answer training instances and 9,250 evaluation samples, spanning six spatial categories: orientation, depth estimation, pixel-level localization, pairwise distance, lateral ordering, and front-behind relations. We benchmark leading general-purpose VLMs, including GPT, Gemini, and Qwen, revealing persistent limitations in fine-grained spatial understanding. To address these deficiencies, we go beyond static evaluation and explore whether alignment techniques can improve spatial reasoning performance. Specifically, we propose a reinforcement learning-based alignment scheme leveraging spatially grounded reward signals - capturing both perception-level accuracy (location) and reasoning consistency (logic). We further incorporate final-answer correctness and output-format rewards to guide fine-grained policy adaptation. Our GRPO-aligned variant achieves an overall score of 40.80 in the SURDS benchmark. Notably, it outperforms proprietary systems such as GPT-4o (13.30) and Gemini-2.0-flash (35.71). To our best knowledge, this is the first study to demonstrate that reinforcement learning-based alignment can significantly and consistently enhance the spatial reasoning capabilities of VLMs in real-world driving contexts. We release the SURDS benchmark, evaluation toolkit, and GRPO alignment code through: https://github.com/XiandaGuo/Drive-MLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。