用多模态大模型知识蒸馏提升自动驾驶规划效率与安全
Distilling Multi-modal Large Language Models for Autonomous Driving
- 通过设计特殊代理任务，将多模态大模型知识蒸馏到视觉规划器
- 使视觉规划器在长尾场景下轨迹误差降低44%，碰撞率下降80%
- 推理时无需大模型，兼顾高效性与强泛化能力，适合部署落地
自动驾驶需在关键的“长尾”场景中实现安全运动规划。近期端到端系统采用大语言模型（LLM）作为规划器以提升对罕见事件的泛化能力，但测试时使用LLM带来高计算开销。为此，我们提出DiMA，一种端到端自动驾驶系统，在保持无LLM（或基于视觉）规划器效率的同时，利用LLM的世界知识。DiMA通过一组专门设计的代理任务，将多模态LLM的信息蒸馏至视觉基端到端规划器。在联合训练策略下，两个网络共享一个场景编码器，生成语义上合理且与最终规划目标对齐的结构化表示。值得注意的是，推理时无需使用LLM，即可实现鲁棒规划而不牺牲效率。在DiMA训练后，视觉规划器的L2轨迹误差降低37%，碰撞率减少80%，长尾场景下轨迹误差减少44%。DiMA还在nuScenes规划基准上达到当前最优性能。
原文摘要 · Abstract (English)
Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。