一个统一模型同时完成自动驾驶的感知、预测与规划，效果超越专用模型。
DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- 用多模态大模型融合点云、图像和语言指令，实现4D空间理解
- 仅用0.5B小模型即在nuScenes等数据集上超越主流大模型
- 适合想用单一框架解决自动驾驶全流程任务的研究者
尽管多模态大语言模型在多个领域展现强大能力，但在生成精细3D感知与预测输出方面，其在自动驾驶中的应用仍不充分。本文提出DrivePI，一种新型的空间感知4D MLLM，作为统一的视觉-语言-动作（VLA）框架，兼容视觉-动作（VA）模型。该方法通过端到端优化，联合执行空间理解、3D感知（即3D占用）、预测（即占用流）与规划（即动作输出）。为获取精确几何信息与丰富视觉外观，方法在统一的MLLM架构中整合点云、多视角图像与语言指令。进一步开发数据引擎，生成文本-占用与文本-流问答对以支持4D空间理解。值得注意的是，仅使用0.5B的Qwen2.5模型作为骨干，DrivePI作为单一统一模型，在性能上匹配或超越现有VLA模型与专用VA模型。具体而言，相比VLA模型，DrivePI在nuScenes-QA上平均准确率高出OpenDriveVLA-7B 2.5%，在nuScenes上碰撞率从0.37%降至0.11%，降低70%；相较于专用VA模型，其在OpenOcc上3D占用任务提升10.3 RayIoU，占用流任务的mAVE从0.591降至0.509，规划任务的L2误差从0.72m降至0.49m，降低32%。代码将公开于https://github.com/happinesslz/DrivePI。
原文摘要 · Abstract (English)
Although multi-modal large language models (MLLMs) have shown strong capabilities across diverse domains, their application in generating fine-grained 3D perception and prediction outputs in autonomous driving remains underexplored. In this paper, we propose DrivePI, a novel spatial-aware 4D MLLM that serves as a unified Vision-Language-Action (VLA) framework that is also compatible with vision-action (VA) models. Our method jointly performs spatial understanding, 3D perception (i.e., 3D occupancy), prediction (i.e., occupancy flow), and planning (i.e., action outputs) in parallel through end-to-end optimization. To obtain both precise geometric information and rich visual appearance, our approach integrates point clouds, multi-view images, and language instructions within a unified MLLM architecture. We further develop a data engine to generate text-occupancy and text-flow QA pairs for 4D spatial understanding. Remarkably, with only a 0.5B Qwen2.5 model as MLLM backbone, DrivePI as a single unified model matches or exceeds both existing VLA models and specialized VA models. Specifically, compared to VLA models, DrivePI outperforms OpenDriveVLA-7B by 2.5% mean accuracy on nuScenes-QA and reduces collision rate by 70% over ORION (from 0.37% to 0.11%) on nuScenes. Against specialized VA models, DrivePI surpasses FB-OCC by 10.3 RayIoU for 3D occupancy on OpenOcc, reduces the mAVE from 0.591 to 0.509 for occupancy flow on OpenOcc, and achieves 32% lower L2 error than VAD (from 0.72m to 0.49m) for planning on nuScenes. Code will be available at https://github.com/happinesslz/DrivePI
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。