用扩散模型实现端到端自动驾驶,速度更快、更安全。
ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
- 采用掩码扩散模型并行生成完整驾驶决策序列
- 在nuScenes上推理速度更快,失败率接近零
- 支持双向推理,适合动态安全场景
基于视觉语言模型(VLM)的端到端自动驾驶系统展现出巨大潜力,但其依赖自回归架构带来高延迟和无法双向推理的问题,难以满足实时安全场景需求。为此,我们提出ViLaD,一种大型视觉语言扩散框架,实现范式转变。ViLaD采用掩码扩散模型,可并行生成完整的驾驶决策序列,显著降低计算延迟。其架构支持双向推理,能同时考虑过去与未来信息,并支持由易到难的渐进生成以逐步提升决策质量。我们在nuScenes数据集上进行综合实验,结果表明,ViLaD在规划准确率和推理速度上均优于现有自回归VLM基线,且故障率接近零。此外,通过在真实自动驾驶车辆上完成交互式泊车任务,验证了该框架的实际可行性与有效性。
原文摘要 · Abstract (English)
End-to-end autonomous driving systems built on Vision Language Models (VLMs) have shown significant promise, yet their reliance on autoregressive architectures introduces some limitations for real-world applications. The sequential, token-by-token generation process of these models results in high inference latency and cannot perform bidirectional reasoning, making them unsuitable for dynamic, safety-critical environments. To overcome these challenges, we introduce ViLaD, a novel Large Vision Language Diffusion (LVLD) framework for end-to-end autonomous driving that represents a paradigm shift. ViLaD leverages a masked diffusion model that enables parallel generation of entire driving decision sequences, significantly reducing computational latency. Moreover, its architecture supports bidirectional reasoning, allowing the model to consider both past and future simultaneously, and supports progressive easy-first generation to iteratively improve decision quality. We conduct comprehensive experiments on the nuScenes dataset, where ViLaD outperforms state-of-the-art autoregressive VLM baselines in both planning accuracy and inference speed, while achieving a near-zero failure rate. Furthermore, we demonstrate the framework's practical viability through a real-world deployment on an autonomous vehicle for an interactive parking task, confirming its effectiveness and soundness for practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。