用知识图谱增强大模型,让自动驾驶更准确回答细节问题。
KLDrive: Fine-Grained 3D Scene Reasoning for Autonomous Driving based on Knowledge Graph
- 构建能量驱动的知识图谱融合多源信息,确保场景事实可靠。
- 在NuScenes-QA和GVQA上分别达到65.04%准确率和42.45分SPICE得分。
- 适合需要高精度场景推理的自动驾驶系统研发与评测。
自动驾驶需对细粒度3D场景事实进行可靠推理。基于多模态驾驶观测的细粒度问答是评估该能力的自然方式,但现有感知流程和面向驾驶的大语言模型仍存在事实不可靠、幻觉、推理不透明及高度依赖特定任务训练等问题。本文提出KLDrive,首个基于知识图谱增强的大型语言模型推理框架,用于自动驾驶中的细粒度问答。该框架设计两个紧密耦合组件:基于能量的场景事实构建模块,将多源证据整合为可信的场景知识图谱;以及在显式结构约束下,于受限动作空间执行事实导向推理的LLM代理。通过结构化提示与少样本上下文示例结合,框架无需大量特定任务微调即可适应多种推理任务。在两个大规模自动驾驶问答基准上的实验表明,KLDrive优于现有最先进方法,在NuScenes-QA上实现65.04%的整体准确率,在GVQA上取得42.45的最高SPICE分数。在最复杂的计数任务中,相较最强基线提升46.01个百分点,显著降低幻觉并验证了可靠事实构建与显式推理结合的优势。
原文摘要 · Abstract (English)
Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。