用外部知识提升自动驾驶模型在罕见场景的决策能力
VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving
- 将视觉输入转为时空语义图，过滤噪声并增强可检索性
- 通过图对齐匹配实现精准知识召回，驱动轨迹生成得分89.12
- 适合研究端到端自动驾驶与知识增强模型的学者
视觉-语言-动作（VLA）模型已成为端到端自动驾驶的有前景范式，但其依赖隐式参数化知识，在长尾场景中泛化能力受限。检索增强生成（RAG）可通过访问外部专家先验提供解决方案，但标准视觉检索存在高延迟和语义模糊问题。为此，我们提出VLADriver-RAG框架，将规划建立在显式、结构感知的历史知识基础上。具体而言，通过视觉到场景机制将感知输入抽象为时空语义图，有效过滤视觉噪声；为确保检索相关性，采用场景对齐嵌入模型，利用图DTW度量对齐，优先考虑内在拓扑一致性而非表面视觉相似性。这些检索到的先验在基于查询的VLA主干中融合，生成精确且解耦的轨迹。在Bench2Drive基准上的大量实验确立了新基准，驾驶得分达到89.12。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving, yet their reliance on implicit parametric knowledge limits generalization in long-tail scenarios. While Retrieval-Augmented Generation (RAG) offers a solution by accessing external expert priors, standard visual retrieval suffers from high latency and semantic ambiguity. To address these challenges, we propose \textbf{VLADriver-RAG}, a framework that grounds planning in explicit, structure-aware historical knowledge. Specifically, we abstract sensory inputs into spatiotemporal semantic graphs via a \textit{Visual-to-Scenario} mechanism, effectively filtering visual noise. To ensure retrieval relevance, we employ a \textit{Scenario-Aligned Embedding Model} that utilizes Graph-DTW metric alignment to prioritize intrinsic topological consistency over superficial visual similarity. These retrieved priors are then fused within a query-based VLA backbone to synthesize precise, disentangled trajectories. Extensive experiments on the Bench2Drive benchmark establish a new state-of-the-art, achieving a Driving Score of 89.12.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。