让自动驾驶模型像人一样逐步推理,提升决策准确性。
MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving
- 分三步模拟人类思考:理解语义、想象场景变化、规划轨迹。
- 在nuScenes和Bench2Drive上表现优于现有方法,闭环评估效果显著。
- 自动生成对齐数据并用强化学习优化推理过程,适合自动驾驶研发者。
视觉语言模型(VLM)具备强大推理能力,有望用于端到端自动驾驶系统。链式思维(CoT)作为其主流推理策略面临挑战:传统文本CoT存在语义空间与轨迹物理空间的鸿沟。近期方法虽用未来图像替代文本进行CoT,但缺乏明确的规划目标引导,难以生成准确的场景演化图像。为此,我们提出MindDriver,一种渐进式多模态推理框架,使VLM能模仿人类逐步思考过程。该框架包含语义理解、语义到物理空间的想象、以及物理空间的轨迹规划。为实现推理过程对齐,我们设计了反馈引导的自动数据标注流程,生成对齐的多模态推理训练数据;同时提出渐进式强化微调方法,通过分层奖励学习优化对齐。MindDriver在nuScenes开环测试和Bench2Drive闭环测试中均表现出色。代码已开源:https://github.com/hotdogcheesewhite/MindDriver。
原文摘要 · Abstract (English)
Vision-Language Models (VLM) exhibit strong reasoning capabilities, showing promise for end-to-end autonomous driving systems. Chain-of-Thought (CoT), as VLM's widely used reasoning strategy, is facing critical challenges. Existing textual CoT has a large gap between text semantic space and trajectory physical space. Although the recent approach utilizes future image to replace text as CoT process, it lacks clear planning-oriented objective guidance to generate images with accurate scene evolution. To address these, we innovatively propose MindDriver, a progressive multimodal reasoning framework that enables VLM to imitate human-like progressive thinking for autonomous driving. MindDriver presents semantic understanding, semantic-to-physical space imagination, and physical-space trajectory planning. To achieve aligned reasoning processes in MindDriver, we develop a feedback-guided automatic data annotation pipeline to generate aligned multimodal reasoning training data. Furthermore, we develop a progressive reinforcement fine-tuning method to optimize the alignment through progressive high- level reward-based learning. MindDriver demonstrates superior performance in both nuScences open-loop and Bench2Drive closed-loop evaluation. Codes are available at https://github.com/hotdogcheesewhite/MindDriver.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。