arXiv:2507.20342cs.AIcs.RO2025-07中稿 · ACM MM 2025被引 6

用视觉语言模型增强自动驾驶规划,让系统更懂复杂路况。

VLMPlanner: Integrating Visual Language Models with Motion Planning

论文配图:VLMPlanner: Integrating Visual Language Models with Motion Planning
图 1 · 摘自论文原文
  • 结合视觉语言模型与实时规划器,直接处理多视角图像。
  • 在nuPlan上实现优于基线的轨迹生成,复杂场景表现突出。
  • 动态调整推理频率,兼顾效率与安全,适合真实驾驶场景。

将大语言模型(LLMs)融入自动驾驶运动规划已成为新方向,可提升可解释性、可控性和罕见场景下的泛化能力。然而现有方法多依赖抽象感知或地图输入,忽略了细粒度道路线索、事故残迹或突发障碍等关键视觉信息,影响复杂环境下的决策鲁棒性。为此,我们提出VLMPlanner,一种融合学习型实时规划器与视觉语言模型(VLM)的混合框架。该VLM通过多视角图像捕捉丰富细节,并利用常识推理能力指导实时规划器生成安全轨迹。此外,我们设计了上下文自适应推理门控机制(CAI-Gate),根据场景复杂度动态调节VLM推理频率,实现性能与效率的最佳平衡。我们在大规模、挑战性强的nuPlan基准上评估该方法,实验表明其在复杂道路条件与动态元素场景中均表现优越。代码将公开。

原文摘要 · Abstract (English)

Integrating large language models (LLMs) into autonomous driving motion planning has recently emerged as a promising direction, offering enhanced interpretability, better controllability, and improved generalization in rare and long-tail scenarios. However, existing methods often rely on abstracted perception or map-based inputs, missing crucial visual context, such as fine-grained road cues, accident aftermath, or unexpected obstacles, which are essential for robust decision-making in complex driving environments. To bridge this gap, we propose VLMPlanner, a hybrid framework that combines a learning-based real-time planner with a vision-language model (VLM) capable of reasoning over raw images. The VLM processes multi-view images to capture rich, detailed visual information and leverages its common-sense reasoning capabilities to guide the real-time planner in generating robust and safe trajectories. Furthermore, we develop the Context-Adaptive Inference Gate (CAI-Gate) mechanism that enables the VLM to mimic human driving behavior by dynamically adjusting its inference frequency based on scene complexity, thereby achieving an optimal balance between planning performance and computational efficiency. We evaluate our approach on the large-scale, challenging nuPlan benchmark, with comprehensive experimental results demonstrating superior planning performance in scenarios with intricate road conditions and dynamic elements. Code will be available.

自动驾驶视觉语言模型运动规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。