用视觉语言模型实现实时自动驾驶，融合地图与目标点提升决策准确率。
Vision-Language Cross-Attention for Real-Time Autonomous Driving
- 通过目标中心交叉注意力融合图像、地图与导航点，实现端到端驾驶决策。
- 在MD-NEX基准上成功率95%、碰撞率减半，优于现有方法15%。
- 适合自动驾驶研发者，支持可解释性输出，兼顾效率与透明度。
自动驾驶汽车需要几何精度与语义理解来应对复杂环境，但现有系统通常将二者分离处理。本文提出XYZ-Drive，一种单一的视觉-语言模型，输入前视摄像头画面、25m×25m俯视地图及下一个目的地坐标，输出转向角与速度指令。采用轻量级目标中心交叉注意力层，使目标点令牌突出相关图像与地图区域，支持动作与文本解释生成，随后融合特征输入部分微调的LLaMA-3.2 11B模型。在MD-NEX Outdoor-Driving基准上，XYZ-Drive达成95%成功率和0.80的路径长度加权成功率（SPL），较PhysNav-DG提升15%，碰撞数减半。仅使用单分支结构显著提升效率。十六组消融实验表明：移除任一模态（视觉、目标点、地图）会导致成功率下降最多11%，验证其互补性；用简单拼接替代目标中心注意力使性能下降3%，说明基于查询的融合更有效注入地图知识；保持Transformer冻结则损失5%，表明针对自动驾驶任务微调至关重要；将地图分辨率从10厘米降至40厘米导致车道线模糊，碰撞率上升。总体表明，意图与地图布局的早期、令牌级融合，能实现精准、可解释且实时的驾驶行为。
原文摘要 · Abstract (English)
Autonomous cars need geometric accuracy and semantic understanding to navigate complex environments, yet most stacks handle them separately. We present XYZ-Drive, a single vision-language model that reads a front-camera frame, a 25m $\times$ 25m overhead map, and the next waypoint, then outputs steering and speed. A lightweight goal-centered cross-attention layer lets waypoint tokens highlight relevant image and map patches, supporting both action and textual explanations, before the fused tokens enter a partially fine-tuned LLaMA-3.2 11B model. On the MD-NEX Outdoor-Driving benchmark XYZ-Drive attains 95% success and 0.80 Success weighted by Path Length (SPL), surpassing PhysNav-DG by 15%. and halving collisions, all while significantly improving efficiency by using only a single branch. Sixteen ablations explain the gains. Removing any modality (vision, waypoint, map) drops success by up to 11%, confirming their complementary roles and rich connections. Replacing goal-centered attention with simple concatenation cuts 3% in performance, showing query-based fusion injects map knowledge more effectively. Keeping the transformer frozen loses 5%, showing the importance of fine-tuning when applying VLMs for specific tasks such as autonomous driving. Coarsening map resolution from 10 cm to 40 cm blurs lane edges and raises crash rate. Overall, these results demonstrate that early, token-level fusion of intent and map layout enables accurate, transparent, real-time driving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。