用视觉语言模型结合先验知识，提升自动驾驶轨迹预测精度
KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
- 通过时序-空间融合编码器与检索增强生成，融合场景知识
- 在nuScenes数据集上实现最优开环预测性能
- 适合关注自动驾驶安全与可解释性的研究者
准确的短时轨迹预测对自动驾驶的安全可靠至关重要。现有视觉语言模型（VLM）常无法充分理解驾驶场景，生成不可靠轨迹。本文提出KEPT框架，直接从连续前视驾驶图像中预测自车轨迹。KEPT采用自监督训练的时序频率-空间融合（TFSF）视频编码器，并结合k-means与HNSW的检索增强生成（RAG）管道，将检索到的先验知识融入思维链（CoT）提示并施加显式规划约束；同时采用三阶段微调策略对VLM主干进行优化，增强空间感知与轨迹预测能力。在nuScenes数据集上的评估表明，KEPT优于基线方法，取得最佳开环性能。消融实验验证了微调阶段、RAG的Top-K值、不同检索策略、视觉编码器及VLM主干的有效性。结果表明，KEPT为实现可信轨迹预测提供了一种高效的数据利用方案。
原文摘要 · Abstract (English)
Accurate short-horizon trajectory prediction is crucial for safe and reliable autonomous driving. However, existing vision-language models (VLMs) often fail to accurately understand driving scenes and generate trustworthy trajectories. To address this challenge, this paper introduces KEPT, a knowledge-enhanced VLM framework that predicts ego trajectories directly from consecutive front-view driving frames. KEPT integrates a temporal frequency-spatial fusion (TFSF) video encoder, which is trained via self-supervised learning with hard-negative mining, with a k-means & HNSW retrieval-augmented generation (RAG) pipeline. Retrieved prior knowledge is added into chain-of-thought (CoT) prompts with explicit planning constraints, while a triple-stage fine-tuning paradigm aligns the VLM backbone to enhance spatial perception and trajectory prediction capabilities. Evaluated on nuScenes dataset, KEPT achieves the best open-loop performance compared with baseline methods. Ablation studies on fine-tuning stages, Top-K value of RAG, different retrieval strategies, vision encoders, and VLM backbones are conducted to demonstrate the effectiveness of KEPT. These results indicate that KEPT offers a promising, data-efficient way toward trustworthy trajectory prediction in autonomous driving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。