arXiv:2510.02803cs.ROcs.AI2025-10被引 2

VLM在施工区轨迹规划中错误率超68%,新框架通过检索增强提升鲁棒性。

Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving

  • 将历史失败案例转为约束规则,结合检索增强生成新场景轨迹
  • 在ROADWork数据集上误差降低3倍,推理速度仅0.58秒
  • 实车测试验证了框架在真实施工区的实用性

视觉语言模型(VLMs)凭借强大的多模态推理能力,正被多家车企用于提升复杂环境下的自动驾驶规划能力。然而,针对施工区这种布局不规则、交通管控临时且几何结构动态变化的场景,VLM的轨迹规划能力尚未得到系统研究。本文首次对VLM在施工区轨迹规划中的表现进行了系统分析,发现主流VLM在68.0%的情况下无法生成正确轨迹。通过子图挖掘与聚类分析识别出候选失败模式,并经人工验证确认了8种常见故障模式。基于此,我们提出REACT-Drive框架,融合VLM与检索增强生成(RAG)。该框架利用VLM将过往失败案例转化为约束规则和可执行的轨迹规划代码,同时RAG在新场景中检索相似模式以指导轨迹生成。在ROADWork数据集上的实验表明,相较于Qwen2.5-VL基线,REACT-Drive平均位移误差降低约3倍;此外,其推理时间仅为0.58秒,远低于微调方法的17.90秒。进一步在真实世界15个施工区场景的实车测试中,验证了REACT-Drive的强实用性。

原文摘要 · Abstract (English)

Visual Language Models (VLMs), with powerful multimodal reasoning capabilities, are gradually integrated into autonomous driving by several automobile manufacturers to enhance planning capability in challenging environments. However, the trajectory planning capability of VLMs in work zones, which often include irregular layouts, temporary traffic control, and dynamically changing geometric structures, is still unexplored. To bridge this gap, we conduct the \textit{first} systematic study of VLMs for work zone trajectory planning, revealing that mainstream VLMs fail to generate correct trajectories in $68.0%$ of cases. To better understand these failures, we first identify candidate patterns via subgraph mining and clustering analysis, and then confirm the validity of $8$ common failure patterns through human verification. Building on these findings, we propose REACT-Drive, a trajectory planning framework that integrates VLMs with Retrieval-Augmented Generation (RAG). Specifically, REACT-Drive leverages VLMs to convert prior failure cases into constraint rules and executable trajectory planning code, while RAG retrieves similar patterns in new scenarios to guide trajectory generation. Experimental results on the ROADWork dataset show that REACT-Drive yields a reduction of around $3\times$ in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL. In addition, REACT-Drive yields the lowest inference time ($0.58$s) compared with other methods such as fine-tuning ($17.90$s). We further conduct experiments using a real vehicle in 15 work zone scenarios in the physical world, demonstrating the strong practicality of REACT-Drive.

自动驾驶VLM轨迹规划RAG

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。