arXiv:2606.29879cs.CVcs.AI2026-06

用分层世界模型引导视觉语言模型,提升自动驾驶规划的精度与前瞻性。

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

论文配图:LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving
图 1 · 摘自论文原文
  • 以VLM输出为意图初稿,通过多层世界模型逐步优化轨迹。
  • 在NAVSIM和NAVSIM-v2上分别达到92.0和89.6的评分,显著优于基线。
  • 适合关注高阶语义与几何精准结合的自动驾驶规划研究者。

视觉语言模型(VLM)为端到端自动驾驶规划提供强大的语义理解与常识推理能力。然而,直接由VLM生成的轨迹通常仅包含粗略驾驶意图,难以实现几何精确、未来感知及多视角一致的规划。为此,我们提出分层世界模型引导的驾驶框架(LWDrive)。LWDrive不将VLM输出视为最终轨迹,而是将其作为意图感知的粗略计划,围绕其扩展多样化候选空间,并通过前瞻级联规划器(FCP)逐层精炼。具体地,引入未来帧生成监督,促使VLM学习前向场景表征,从而将规划相关的预测动态注入其内部隐藏状态。基于这些受世界模型监督的表征,FCP利用VLM多层特征,融合历史时序状态、动作查询表示与当前帧多视角鸟瞰图(BEV)特征,以粗到精的方式精炼候选轨迹。该设计可逐步修正空间位置与运动趋势,同时结合多视角场景线索进行轨迹锚定,并保留大模型产生的高层驾驶意图。最后,评分头对精炼候选进行评估,选择最优轨迹作为最终输出。实验表明,LWDrive在NAVSIM基准上取得92.0分,在NAVSIM-v2上取得89.6分。代码与模型将公开发布。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning. However, trajectories directly generated by VLMs often encode only coarse driving intentions and remain insufficient for geometrically accurate, future-aware, and multi-view-grounded planning. To address these limitations, we develop the Layer-Wise World-Model-Guided Driving framework (LWDrive). LWDrive is a VLM planning framework that refines coarse trajectories through layer-wise world-model guidance. Instead of treating the VLM output as the final trajectory, LWDrive uses it as an intent-aware coarse plan, expands a diverse candidate space around it, and progressively refines the candidates through a Foresight Cascade Planner (FCP). Specifically, we introduce future-frame generation supervision to encourage the VLM to learn forward-looking scene representations, thereby injecting planning-relevant predictive dynamics into its internal hidden states. Built upon these world-model-supervised representations, FCP exploits VLM features across multiple layers and integrates historical temporal states, Action-Query representations, and current-frame multi-view Bird's-Eye-View (BEV) features to refine candidate trajectories in a coarse-to-fine manner. This design enables progressive correction of spatial positions and motion trends while grounding trajectory refinement with multi-view scene cues and preserving the high-level driving intention produced by the large model. Finally, a score head evaluates the refined candidates and selects the best trajectory as the final planning output. Experiments show that LWDrive achieves a score of 92.0 on the NAVSIM benchmark and 89.6 on NAVSIM-v2. Code and models will be made publicly available.

自动驾驶视觉语言模型轨迹规划世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。