用本地大模型增强自动驾驶决策与控制，提升应对突发情况的能力。
Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Models
- 将大语言模型嵌入车载系统，结合预测控制实现智能决策。
- 推理准确率提升10.45%，控制适应性提高52.2%，效率提升10.5倍。
- 适合需要本地化部署、高安全性的自动驾驶研发团队。
通过监督学习训练的神经网络难以应对真实驾驶中的边缘案例，因其无法穷尽所有异常场景。本文提出一种混合架构，将低层模型预测控制（MPC）与本地部署的大语言模型（LLMs）结合，提升决策与人机交互能力。DecisionxLLM模块基于自然语言指令评估机器人状态，确保符合期望驾驶行为；MPCxLLM模块则根据大模型输出调整MPC参数，保持传统MPC的安全性与约束保障的同时实现控制自适应。为支持车载高效部署并摆脱云端依赖，采用检索增强生成（RAG）、低秩适配（LoRA）微调和量化技术。实验表明，该方法使推理准确率提升最高达10.45%，控制适应性提高最多52.2%，计算效率提升最高达10.5倍（tokens/s），验证了其在小型机器人平台上的实时部署可行性。该框架实现了高层决策与底层控制的协同，为知识驱动型自适应自动驾驶系统提供新范式。
原文摘要 · Abstract (English)
Neural Networks (NNs) trained through supervised learning struggle with managing edge-case scenarios common in real-world driving due to the intractability of exhaustive datasets covering all edge-cases, making knowledge-driven approaches, akin to how humans intuitively detect unexpected driving behavior, a suitable complement to data-driven methods. This work proposes a hybrid architecture combining low-level Model Predictive Controller (MPC) with locally deployed Large Language Models (LLMs) to enhance decision-making and Human Machine Interaction (HMI). The DecisionxLLM module evaluates robotic state information against natural language instructions to ensure adherence to desired driving behavior. The MPCxLLM module then adjusts MPC parameters based on LLM-generated insights, achieving control adaptability while preserving the safety and constraint guarantees of traditional MPC systems. Further, to enable efficient on-board deployment and to eliminate dependency on cloud connectivity, we shift processing to the on-board computing platform: We propose an approach that exploits Retrieval Augmented Generation (RAG), Low Rank Adaptation (LoRA) fine-tuning, and quantization. Experimental results demonstrate that these enhancements yield significant improvements in reasoning accuracy by up to 10.45%, control adaptability by as much as 52.2%, and up to 10.5x increase in computational efficiency (tokens/s), validating the proposed framework's practicality for real-time deployment even on down-scaled robotic platforms. This work bridges high-level decision-making with low-level control adaptability, offering a synergistic framework for knowledge-driven and adaptive Autonomous Driving Systems (ADS).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。