arXiv:2505.15925cs.ROcs.AI2025-05被引 19

将大模型推理能力融入自动驾驶系统,实现高效安全决策

VERDI: VLM-Embedded Reasoning for Autonomous Driving

  • 用视觉语言模型生成驾驶推理文本,对齐各模块中间输出
  • 在开环和闭环测试中提升11%轨迹精度,碰撞率降低10%
  • 无需运行大模型即可推理,适合实际部署的自动驾驶系统

当前自动驾驶系统在部分可观测和现实复杂场景下决策能力不足,而人类驾驶员能基于常识做出近似最优判断。现有方法虽尝试通过微调视觉语言模型(VLM)在推理时进行轨迹规划以模仿人类行为,但其70B参数模型每秒仅处理8个标记,需超过160GB内存,且单体结构难以保障安全性。为此,我们提出VERDI:一种训练时将VLM的推理过程与常识知识蒸馏进自动驾驶栈的框架。VERDI通过将感知、预测、规划阶段的中间输出与由VLM生成的解释性文本特征对齐,使模块化可微端到端系统在隐空间内内化结构化推理能力,无需在推理时运行大模型。我们在开环与闭环设置中评估,结果表明,本方法相较无推理嵌入的端到端方案最多提升11%的ℓ₂距离表现,并在闭环HugSim仿真中取得最佳综合驾驶性能,包括10%的非碰撞率提升,同时保持快速推理速度。

原文摘要 · Abstract (English)

While autonomous driving (AD) stacks struggle with decision making under partial observability and real-world complexity, human drivers are capable of applying commonsense reasoning to make near-optimal decisions with limited information. Recent work has attempted to leverage finetuned Vision-Language Models (VLMs) for trajectory planning at inference time to emulate human behavior. Despite their success in benchmark evaluations, these methods are often impractical to deploy (a 70B parameter VLM inference at merely 8 tokens per second requires more than 160G of memory), and their monolithic network structure prohibits safety decomposition. To bridge this gap, we propose VLM-Embedded Reasoning for autonomous DrIving (VERDI), a training-time framework that distills the reasoning process and commonsense knowledge of VLMs into the AD stack. VERDI augments modular differentiable end-to-end (e2e) AD models by aligning intermediate module outputs at the perception, prediction, and planning stages with text features explaining the driving reasoning process produced by VLMs. By encouraging alignment in latent space, VERDI enables the modular AD stack to internalize structured reasoning, without incurring the inference-time costs of large VLMs. We evaluate VERDI in both open-loop and closed-loop settings. Our method outperforms existing end-to-end approaches without embedded reasoning by up to 11% in $\ell_{2}$ distance, and achieves the best overall driving performance in the closed-loop HugSim simulator, including a 10% improvement in Non-Collision Rate, while maintaining fast inference speed.

自动驾驶视觉语言模型推理蒸馏端到端

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。