让视觉语言动作模型学会看对地方，提升自动驾驶规划能力。
Teaching Vision-Language-Action Models What to See and Where to Look
- 用驾驶感知先验增强视觉编码器，明确该看什么
- 通过轨迹引导提示，让模型聚焦可行行驶区域
- 在nuScenes和NAVSIM上达到当前最佳性能，适合自动驾驶研究者
视觉-语言-动作（VLA）模型已成为端到端自动驾驶的有前景范式。但现有VLA训练严重依赖以文本为中心的视觉问答和思维链推理数据，侧重语言推理而非基于行动的规划，导致学习表征虽具语义知识，却缺乏可靠轨迹预测所需的空间依赖关系。我们提出DriveTeach-VLA框架，显式教导VLA模型应关注什么以及看向何处：驾驶感知视觉蒸馏（DVD）向视觉编码器注入驾驶特定的感知先验；2D轨迹引导提示（2D-TGP）提供与可行驾驶轨迹对齐的空间条件。二者共同构成视觉引导学习流程：先确定该看什么（DVD预训练）——再确定看向何处（TGP引导的SFT）——最后如何执行动作（TGP引导的GRPO）。DriveTeach-VLA在NAVSIM和nuScenes数据集上达到当前最优表现。代码已开源：https://github.com/ShivaTeam/DriveTeach-VLA。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。