用统一模型同时处理视觉、语言和驾驶决策，提升自动驾驶效率。
OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models
- 用单一Transformer解码器融合文本生成、目标检测和轨迹预测
- 在nuScenes上实现0.28 L2误差和0.18碰撞率，闭环评估达86.8 PDMS
- 共享预训练注意力结构，推理延迟降低约40%，适合部署
视觉-语言模型（VLM）擅长自回归文本生成，但端到端自动驾驶需要多任务学习与异构输出，如自回归语言生成、并行目标检测和轨迹回归。现有系统常采用独立或级联解码器，导致架构碎片化且主干网络复用受限。本文提出基于预训练VLM的统一自动驾驶框架，将异构解码行为整合进单一Transformer解码器。实验表明，预训练VLM注意力机制在跨模态任务中具有强迁移能力。通过在单个因果解码器内组织视觉与结构化查询标记，结构化输出可自然依赖视觉上下文。文本与结构化输出共享同一注意力主干，实现跨异构任务的稳定联合优化。轨迹规划通过引入结构化轨迹查询在相同因果LLM解码器中完成，使规划任务可共享图像与感知标记的预训练注意力主干。在多个端到端自动驾驶基准测试中，该方法达到领先性能：nuScenes开环评估中L2误差为0.28，碰撞率为0.18；NAVSIM闭环评估得86.8 PDMS。全模型保持多模态生成能力，高效推理模式下延迟降低约40%。代码与模型已开源。
原文摘要 · Abstract (English)
Vision-Language Models(VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (86.8 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency. Code and models are available at https://github.com/Z1zyw/OneDrive
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。