arXiv:2512.10719cs.CV2025-12被引 28

让视觉语言模型更懂空间,提升自动驾驶规划精度。

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

  • 用三维坐标编码替代数字文本,显式注入空间信息
  • 在nuScenes上达到最先进开环性能,闭环得分78.02
  • 适合研究视觉语言模型与自动驾驶融合的开发者

基于视觉语言模型(VLM)的端到端自动驾驶方法因具备强大的通用视觉理解与推理能力而迅速发展。然而我们发现,现有VLM难以理解精细的三维空间关系,而这正是与物理世界交互的基础需求。为此,我们提出SpaceDrive,一种具空间感知能力的VLM驱动驾驶框架,将空间信息作为显式的位置编码(PE)而非文本数字标记,实现语义与空间表示的联合推理。SpaceDrive采用统一的位置编码器处理多视角深度估计、历史车辆状态和文本提示所得的3D坐标。这些3D PEs首先叠加至对应2D视觉标记以增强表征,同时作为任务无关的坐标表示,取代原有基于数字的输入输出形式。该机制使模型能更准确地在空间中定位语义信息,并直接回归轨迹坐标而非逐位生成,从而提升规划准确性。大量实验表明,SpaceDrive在nuScenes数据集上达到最先进的开环表现,在Bench2Drive闭环基准上取得78.02的第二高驾驶得分,优于现有基于VLM的方法。代码已开源:https://github.com/zhenghao2519/SpaceDrive。

原文摘要 · Abstract (English)

End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods. Code is available at: https://github.com/zhenghao2519/SpaceDrive.

自动驾驶视觉语言模型空间感知轨迹规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。