用视觉语言模型提升自动驾驶轨迹规划的推理与泛化能力
LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
- 结合视觉语言模型实现场景理解与思维链推理
- 在NuScenes上误差0.31米，碰撞率仅0.10%
- 适合研究自动驾驶决策与多模态模型融合的读者
轨迹规划是自动驾驶的核心挑战。端到端模型在恶劣天气、复杂路况或人类行为不可预测时表现不佳，主要因泛化能力弱且缺乏少样本适应能力。本文提出LLaViDA，一个基于视觉语言模型（VLM）的驾驶辅助系统，通过对象运动预测、语义定位和思维链推理实现轨迹规划。采用两阶段训练：监督微调后接基于轨迹偏好的优化（TPO），引入回归监督增强场景理解与规划能力，形成强大的‘VLM轨迹规划器’。在NuScenes基准上，其开环轨迹规划任务平均L2误差为0.31米，测试集碰撞率为0.10%，超越现有端到端及基于VLM/LLM的先进方法。
原文摘要 · Abstract (English)
Trajectory planning is a fundamental yet challenging component of autonomous driving. End-to-end planners frequently falter under adverse weather, unpredictable human behavior, or complex road layouts, primarily because they lack strong generalization or few-shot capabilities beyond their training data. We propose LLaViDA, a Large Language Vision Driving Assistant that leverages a Vision-Language Model (VLM) for object motion prediction, semantic grounding, and chain-of-thought reasoning for trajectory planning in autonomous driving. A two-stage training pipeline--supervised fine-tuning followed by Trajectory Preference Optimization (TPO)--enhances scene understanding and trajectory planning by injecting regression-based supervision, produces a powerful "VLM Trajectory Planner for Autonomous Driving." On the NuScenes benchmark, LLaViDA surpasses state-of-the-art end-to-end and other recent VLM/LLM-based baselines in open-loop trajectory planning task, achieving an average L2 trajectory error of 0.31 m and a collision rate of 0.10% on the NuScenes test set. The code for this paper is available at GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。