用视觉语言模型提升自动驾驶安全，训练时用、部署时不用。
DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
- 分双路径处理：静态路径看空间安全，动态路径多帧推理风险。
- 碰撞速度降为1.75km/h，成功率最高，比最强基线提升显著。
- 模型训练用大模型，上线不跑，解决延迟和幻觉问题。
传统强化学习依赖人工设计奖励或稀疏碰撞信号，难以捕捉驾驶所需的丰富上下文信息，导致真实场景中不可避免的危险探索。近期视觉语言模型（VLM）具备出色的语义理解能力，但其高推理延迟和幻觉问题限制了在实时车辆控制中的应用。为此，本文提出DriveVLM-RL，一种受神经科学启发的框架，通过双通路架构将VLM融入强化学习，实现安全且可部署的自动驾驶。受人类大脑习惯性与反思性视觉处理机制启发，该框架将语义奖励学习分解为：基于CLIP对比语言目标的静态路径，用于连续空间安全评估；以及基于轻量检测模型与大视觉语言模型（LVLM）的动态路径，实现注意力门控的多帧语义风险推理。层级奖励融合机制结合车辆状态信息，异步训练流程将昂贵的LVLM推理与环境交互解耦。关键的是，所有VLM组件仅在离线训练阶段使用，部署时完全移除，消除测试时推理延迟。大量实验在CARLA模拟器中表明，DriveVLM-RL显著优于现有先进基线，在避撞与任务成功率上表现最优，碰撞严重度从10.09降至1.75 km/h。演示视频、代码与模型权重已公开：https://zilin-huang.github.io/DriveVLM-RL-website/
原文摘要 · Abstract (English)
Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings. Recent vision-language models (VLMs) offer promising semantic understanding capabilities; however, their high inference latency and susceptibility to hallucination hinder direct application to real-time vehicle control. To address these limitations, this paper proposes DriveVLM-RL, a neuroscience-inspired framework that integrates VLMs into RL through a dual-pathway architecture for safe and deployable autonomous driving. Inspired by the human brain's habitual and deliberative visual processing, DriveVLM-RL decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment via CLIP-based contrasting language goals, and a Dynamic Pathway for attention-gated multi-frame semantic risk reasoning via a lightweight detection model and large VLM (LVLM). A hierarchical reward synthesis mechanism fuses these signals with vehicle state information, while an asynchronous training pipeline decouples expensive LVLM inference from environment interaction. Critically, all VLM components operate exclusively during offline training and are completely removed at deployment, eliminating inference latency at test time. Extensive experiments in the CARLA simulator demonstrate that DriveVLM-RL significantly outperforms state-of-the-art baselines in collision avoidance and task success, attaining the highest success rate while reducing collision severity from 10.09 to 1.75 km/h relative to the strongest VLM-based baseline. The demo video, code, and model checkpoints are available at: https://zilin-huang.github.io/DriveVLM-RL-website/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。