将视觉语言模型与场景生成模型结合,实现自动驾驶的智能规划与想象。
ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- 用视觉语言模型做决策,驱动高精度场景生成,形成闭环推理。
- 在nuScenes和NAVSIM上测试,闭环表现优于现有方法。
- 通过早期停止和轨迹选择提升效率,适合复杂城市驾驶场景。
自动驾驶需在动态复杂环境中实现丰富的上下文理解与精准预测推理。视觉语言模型(VLM)擅长多模态语义理解与行为预测,而驾驶世界模型(DWM)能生成细节丰富且合理的未来场景,用于主动规划。二者结合可互补优势,但面临动作决策与像素级预测连接困难、计算效率低等挑战。本文提出ImagiDrive,一种端到端框架,将基于VLM的驾驶代理与基于DWM的场景想象器融合,构建统一的想象-规划循环:驾驶代理根据多模态输入预测初始轨迹,引导场景想象器生成对应未来场景;这些想象场景再反馈优化驾驶决策。为提升效率与准确性,引入早停机制与轨迹选择策略。在nuScenes与NAVSIM数据集上的大量实验表明,ImagiDrive在开环与闭环条件下均显著优于现有方法。
原文摘要 · Abstract (English)
Autonomous driving requires rich contextual comprehension and precise predictive reasoning to navigate dynamic and complex environments safely. Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge. VLMs provide interpretability and robust action prediction through their ability to understand multi-modal context, while DWMs excel in generating detailed and plausible future driving scenarios essential for proactive planning. Integrating VLMs with DWMs is an intuitive, promising, yet understudied strategy to exploit the complementary strengths of accurate behavioral prediction and realistic scene generation. Nevertheless, this integration presents notable challenges, particularly in effectively connecting action-level decisions with high-fidelity pixel-level predictions and maintaining computational efficiency. In this paper, we propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer to form a unified imagination-and-planning loop. The driving agent predicts initial driving trajectories based on multi-modal inputs, guiding the scene imaginer to generate corresponding future scenarios. These imagined scenarios are subsequently utilized to iteratively refine the driving agent's planning decisions. To address efficiency and predictive accuracy challenges inherent in this integration, we introduce an early stopping mechanism and a trajectory selection strategy. Extensive experimental validation on the nuScenes and NAVSIM datasets demonstrates the robustness and superiority of ImagiDrive over previous alternatives under both open-loop and closed-loop conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。