用新方法让通用模型与专用模型协同驾驶,提升安全性和感知精度。
NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
- 通过三阶段机制实现大模型与专用模型的异步对接。
- 在nuScenes上显著提升规划任务的人类相似度与检测精度。
- 适合追求高安全性的端到端自动驾驶研究者使用。
将通用模型(如大语言模型)与专用驾驶模型结合,有望缓解现有专用模型在数据多样性和模型容量上的挑战。然而,二者特性差异导致系统异步问题。为此,我们提出NetRoller,一种包含三项新机制的适配器,实现通用模型与专用模型的无缝集成。首先,通过早停机制从大模型推理中提取语义丰富且计算高效表示,保留关键驾驶上下文信息同时保持低开销;其次,采用可学习查询嵌入、无意义嵌入和位置层嵌入,实现跨模态稳定转换;最后,利用计算高效的查询偏移与特征偏移机制,通过少量训练轮次提升专用模型性能。基于该设计,专用模型可在原生频率下运行并维持大模型的情境感知能力。在nuScenes数据集上的实验表明,通过NetRoller整合通用模型显著提升了规划任务中的人类相似度与安全性,并在检测与地图构建任务中实现显著精度提升。
原文摘要 · Abstract (English)
Integrating General Models (GMs) such as Large Language Models (LLMs), with Specialized Models (SMs) in autonomous driving tasks presents a promising approach to mitigating challenges in data diversity and model capacity of existing specialized driving models. However, this integration leads to problems of asynchronous systems, which arise from the distinct characteristics inherent in GMs and SMs. To tackle this challenge, we propose NetRoller, an adapter that incorporates a set of novel mechanisms to facilitate the seamless integration of GMs and specialized driving models. Specifically, our mechanisms for interfacing the asynchronous GMs and SMs are organized into three key stages. NetRoller first harvests semantically rich and computationally efficient representations from the reasoning processes of LLMs using an early stopping mechanism, which preserves critical insights on driving context while maintaining low overhead. It then applies learnable query embeddings, nonsensical embeddings, and positional layer embeddings to facilitate robust and efficient cross-modality translation. At last, it employs computationally efficient Query Shift and Feature Shift mechanisms to enhance the performance of SMs through few-epoch fine-tuning. Based on the mechanisms formalized in these three stages, NetRoller enables specialized driving models to operate at their native frequencies while maintaining situational awareness of the GM. Experiments conducted on the nuScenes dataset demonstrate that integrating GM through NetRoller significantly improves human similarity and safety in planning tasks, and it also achieves noticeable precision improvements in detection and mapping tasks for end-to-end autonomous driving. The code and models are available at https://github.com/Rex-sys-hk/NetRoller .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。