arXiv:2604.21479cs.CV2026-04中稿 · publication at IEE…

用冻结大模型+地图信息预测车辆轨迹,提升自动驾驶感知能力。

Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Prediction

论文配图:Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Prediction
图 1 · 摘自论文原文
  • 用冻结大模型做时空推理,结合轨迹和高精地图特征
  • 引入轻量CNN编码地图,通过适配器转为大模型可处理格式
  • 无需微调即可兼容多模型,适合研究大模型在自动驾驶中的应用

大型语言模型(LLMs)近期展现出强大的推理能力,受到自动驾驶(AD)领域越来越多的关注。然而,安全应用LLMs于AD感知与预测仍需深入理解动态交通参与者与静态道路基础设施。为此,本文提出一种框架,评估LLMs对动态交通主体行为及道路拓扑结构的理解能力。该框架利用冻结的LLMs作为推理引擎,采用交通编码器从观测轨迹中提取空间级场景特征,同时使用轻量级卷积神经网络(CNN)编码局部高精地图(HD maps)。为评估LLMs内在推理能力,提取的场景特征经由重编程适配器转换为LLM兼容的标记。通过将预测任务交由LLMs完成,仅使用简单的线性解码器输出未来轨迹。该框架实现了对多模态信息(尤其是地图语义)影响的定量分析,支持冻结LLMs的无缝集成且适应成本极低,展现出跨多种LLM架构的强大泛化能力,并提供统一的模型评估平台。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently demonstrated strong reasoning capabilities and attracted increasing research attention in the field of autonomous driving (AD). However, safe application of LLMs on AD perception and prediction still requires a thorough understanding of both the dynamic traffic agents and the static road infrastructure. To this end, this study introduces a framework to evaluate the capability of LLMs in understanding the behaviors of dynamic traffic agents and the topology of road networks. The framework leverages frozen LLMs as the reasoning engine, employing a traffic encoder to extract spatial-level scene features from observed trajectories of agents, while a lightweight Convolutional Neural Network (CNN) encodes the local high-definition (HD) maps. To assess the intrinsic reasoning ability of LLMs, the extracted scene features are then transformed into LLM-compatible tokens via a reprogramming adapter. By residing the prediction burden with the LLMs, a simpler linear decoder is applied to output future trajectories. The framework enables a quantitative analysis of the influence of multi-modal information, especially the impact of map semantics on trajectory prediction accuracy, and allows seamless integration of frozen LLMs with minimal adaptation, thereby demonstrating strong generalizability across diverse LLM architectures and providing a unified platform for model evaluation.

自动驾驶大模型轨迹预测地图感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。