用轨迹信息增强大模型，提升自动驾驶场景理解能力。
Tracking Meets Large Multimodal Models for Driving Scenario Understanding
- 将3D轨迹数据通过编码器融入大模型，补充图像缺失的时空信息。
- 在DriveLM-nuScenes上提升9.5%准确率，ChatGPT评分增7.04点。
- 适合关注自动驾驶感知与决策的科研人员和工程师。
大型多模态模型（LMMs）在自动驾驶研究中日益重要，在多个新兴基准上展现出良好性能，尤其在感知、规划和预测任务中表现突出。然而，现有方法大多忽视3D空间与时间维度，主要依赖图像数据，导致在动态驾驶环境中效果受限。本文提出将跟踪信息作为额外输入，以恢复图像中未充分捕捉的3D时空细节。我们设计了一种新颖的轨迹嵌入方法，通过轨迹编码器将跟踪数据融入LMM，丰富视觉查询中的时空线索，同时避免处理长视频或大规模3D输入带来的计算开销。此外，采用自监督预训练策略对轨迹编码器进行训练，为LMM提供额外上下文信息，显著提升其在感知、规划与预测任务中的表现。实验表明，在DriveLM-nuScenes基准上，本方法相比基线模型准确率提升9.5%，ChatGPT评分增加7.04点，综合得分提高9.4%；在DriveLM-CARLA上最终得分提升3.7%。代码已开源。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have recently gained prominence in autonomous driving research, showcasing promising capabilities across various emerging benchmarks. LMMs specifically designed for this domain have demonstrated effective perception, planning, and prediction skills. However, many of these methods underutilize 3D spatial and temporal elements, relying mainly on image data. As a result, their effectiveness in dynamic driving environments is limited. We propose to integrate tracking information as an additional input to recover 3D spatial and temporal details that are not effectively captured in the images. We introduce a novel approach for embedding this tracking information into LMMs to enhance their spatiotemporal understanding of driving scenarios. By incorporating 3D tracking data through a track encoder, we enrich visual queries with crucial spatial and temporal cues while avoiding the computational overhead associated with processing lengthy video sequences or extensive 3D inputs. Moreover, we employ a self-supervised approach to pretrain the tracking encoder to provide LMMs with additional contextual information, significantly improving their performance in perception, planning, and prediction tasks for autonomous driving. Experimental results demonstrate the effectiveness of our approach, with a gain of 9.5% in accuracy, an increase of 7.04 points in the ChatGPT score, and 9.4% increase in the overall score over baseline models on DriveLM-nuScenes benchmark, along with a 3.7% final score improvement on DriveLM-CARLA. Our code is available at https://github.com/mbzuai-oryx/TrackingMeetsLMM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。