NovaDrive用单分支模型融合视觉与语言，让自动驾驶更安全高效。
Goal-Based Vision-Language Driving
- 单分支架构统一处理图像、地图、激光雷达和文本指令
- 成功率84%（+4%），碰撞率降至1.2%（-1.4%）
- 无需循环记忆，适合实时部署，可推广至其他智能体场景
自动驾驶需在毫秒级响应复杂路况，同时理解道路结构与交通意图。我们提出NovaDrive，一种单分支视觉语言架构，可同时处理前视图像、高清地图块、激光雷达深度图和文本路点。通过轻量级双阶段交叉注意力模块，先对齐路点与地图，再细化图像与深度片段的关注。结合新型平滑性损失以抑制急转弯和速度突变，该设计无需循环记忆。我们在110亿参数的LLaMA-3.2视觉语言主干网络上微调顶层15层，实现实时推理。在nuScenes/Waymo子集的MD-NEX Outdoor基准上，NovaDrive将成功率提升至84%（+4%），路径效率（SPL）达0.66（+0.11），碰撞频率从2.6%降至1.2%（-1.4%）。消融实验表明，路点标记、部分微调及交叉注意力融合是主要贡献因素。此外，因路径更短，能耗更低，推动驾驶系统更轻量化、易更新。该方法亦可扩展至其他具身智能领域。
原文摘要 · Abstract (English)
Autonomous vehicles must react in milliseconds while reasoning about road geometry and traffic intent to navigate complex situations. We introduce NovaDrive, a single-branch vision-language architecture that processes front-camera images, HD-map tiles, LiDAR depth, and textual waypoints in a single branch. A lightweight, two-stage cross-attention block first aligns waypoint tokens with the HD map, then refines attention over fine-grained image and depth patches. Coupled with a novel smoothness loss that discourages abrupt steering and speed changes, this design eliminates the need for recurrent memory. We fine-tune the top 15 layers of an 11B LLaMA-3.2 vision-language backbone, enabling real-time inference. On the nuScenes / Waymo subset of the MD-NEX Outdoor benchmark, NovaDrive raises success rate to 84% (+4%), boosts path-efficiency (SPL) to 0.66 (+0.11), and reduces collision frequency from 2.6% to 1.2% (-1.4%) relative to the previous state-of-the-art. Our ablations confirm that waypoint tokens, partial VLM fine-tuning, and the cross-attention fusion each contribute the most to these gains. Beyond safety, NovaDrive's shorter routes (resulting from the novel smoothness loss) translate to lower fuel or battery usage, pointing toward leaner, more easily updated driving stacks. NovaDrive can be extended to other embodied-AI domains as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。