让汽车听懂乘客的自然语言指令，实现更安全的智能导航。
Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
- 用视觉-语言-动作模型，将乘客口语指令转化为驾驶轨迹。
- 在849个真实场景中，指令引导使平均轨迹误差降低98.7%。
- 适合研究人机交互、自动驾驶决策与多模态智能系统者参考。
以乘客语言指导路径规划的指令驱动驾驶，要求车辆在运动前理解意图。然而，以往多数指令跟随规划器依赖仿真或固定命令词汇表，限制了真实世界泛化能力。doScenes是首个将自由形式指令（含指代性）与nuScenes真实轨迹关联的真实世界数据集，支持指令条件下的规划。本文将开源的基于多模态大模型（MLLM）的OpenEMMA框架适配此设定，提出可在doScenes上复现的指令条件规划基线，并研究人类指令提示对预测行为的影响。我们把doScenes指令作为乘客风格提示嵌入OpenEMMA的视觉-语言接口，实现生成前的语言条件化。在849个标注场景上使用ADE评估，发现指令条件化显著提升鲁棒性，避免了基线极端失败，使均值ADE减少98.7%。剔除异常值后，指令仍影响轨迹对齐，表述良好的提示可使ADE改善最多5.1%。基于分析，我们探讨了适用于OpenEMMA框架的“优质”指令特征。项目提供评估提示与脚本，建立可复现的指令感知规划基准。GitHub: https://github.com/Mi3-Lab/doScenes-VLM-Planning
原文摘要 · Abstract (English)
Instruction-grounded driving, where passenger language guides trajectory planning, requires vehicles to understand intent before motion. However, most prior instruction-following planners rely on simulation or fixed command vocabularies, limiting real-world generalization. doScenes, the first real-world dataset linking free-form instructions (with referentiality) to nuScenes ground-truth motion, enables instruction-conditioned planning. In this work, we adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework that ingests front-camera views and ego-state and outputs 10-step speed-curvature trajectories, to this setting, presenting a reproducible instruction-conditioned baseline on doScenes and investigate the effects of human instruction prompts on predicted driving behavior. We integrate doScenes directives as passenger-style prompts within OpenEMMA's vision-language interface, enabling linguistic conditioning before trajectory generation. Evaluated on 849 annotated scenes using ADE, we observe that instruction conditioning substantially improves robustness by preventing extreme baseline failures, yielding a 98.7% reduction in mean ADE. When such outliers are removed, instructions still influence trajectory alignment, with well-phrased prompts improving ADE by up to 5.1%. We use this analysis to discuss what makes a "good" instruction for the OpenEMMA framework. We release the evaluation prompts and scripts to establish a reproducible baseline for instruction-aware planning. GitHub: https://github.com/Mi3-Lab/doScenes-VLM-Planning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。