将视觉语言模型与端到端驾驶结合,提升复杂场景下的规划能力。
Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

- 分层设计:语言模型做高层决策,端到端模型生成精确轨迹
- 在nuScenes上规划误差降低27.12%,碰撞率下降33.33%
- 适合需要强推理与泛化能力的自动驾驶研发者
端到端自动驾驶虽具备强大规划能力,但在复杂罕见场景中仍受限于常识缺失。大型视觉语言模型(LVLM)在场景理解与推理方面表现优异。本文提出Senna系统,融合LVLM(Senna-VLM)与端到端模型(Senna-E2E)。Senna将高层规划与低层轨迹预测解耦:Senna-VLM以自然语言输出规划决策,Senna-E2E负责精确轨迹生成。Senna-VLM采用多图像编码与多视角提示实现高效场景理解,并引入面向规划的问答对及三阶段训练策略,增强推理能力并保留常识。在DriveX预训练、nuScenes微调下,实验显示其平均规划误差减少27.12%,碰撞率降低33.33%,达到当前最优规划性能。Senna展现出跨场景泛化与可迁移性,对实现全自动驾驶具有重要意义。代码与模型将开源。
原文摘要 · Abstract (English)
End-to-end autonomous driving demonstrates strong planning capabilities with large-scale data but still struggles in complex, rare scenarios due to limited commonsense. In contrast, Large Vision-Language Models (LVLMs) excel in scene understanding and reasoning. The path forward lies in merging the strengths of both approaches. Previous methods using LVLMs to predict trajectories or control signals yield suboptimal results, as LVLMs are not well-suited for precise numerical predictions. This paper presents Senna, an autonomous driving system combining an LVLM (Senna-VLM) with an end-to-end model (Senna-E2E). Senna decouples high-level planning from low-level trajectory prediction. Senna-VLM generates planning decisions in natural language, while Senna-E2E predicts precise trajectories. Senna-VLM utilizes a multi-image encoding approach and multi-view prompts for efficient scene understanding. Besides, we introduce planning-oriented QAs alongside a three-stage training strategy, which enhances Senna-VLM's planning performance while preserving commonsense. Extensive experiments on two datasets show that Senna achieves state-of-the-art planning performance. Notably, with pre-training on a large-scale dataset DriveX and fine-tuning on nuScenes, Senna significantly reduces average planning error by 27.12% and collision rate by 33.33% over model without pre-training. We believe Senna's cross-scenario generalization and transferability are essential for achieving fully autonomous driving. Code and models will be released at https://github.com/hustvl/Senna.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。