arXiv:2503.12772cs.CVcs.AI2025-03ICCV被引 31

构建100万组多视角驾驶问答数据,评估大模型对交通场景的理解能力。

NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models

  • 设计九类子任务,覆盖道路感知、空间关系与自身视角推理。
  • 提出BEV-LLM模型,在六项任务中超越现有模型。
  • 公开百万级数据集,助力多视角驾驶理解研究。

多模态大语言模型在多个领域表现优异,但在驾驶场景理解方面仍存短板。本文提出NuPlanQA-Eval,一个面向多视角驾驶场景理解的多模态评估基准。为支持模型在多视角场景中的泛化能力,我们构建了包含100万组真实世界视觉问答对的NuPlanQA-1M数据集。基于上下文感知分析,将数据集分为九个子任务,涵盖道路环境感知、空间关系识别和自身中心推理三类核心能力。同时提出BEV-LLM,将多视角图像的鸟瞰图(BEV)特征融入大模型。实验表明,现有模型在驾驶场景感知与自身视角推理上存在明显不足;而BEV-LLM在六个子任务中表现更优。结果验证了BEV特征对多视角建模的提升作用,并指出未来需进一步优化的方向。相关数据集已开源,详见https://github.com/sungyeonparkk/NuPlanQA。

原文摘要 · Abstract (English)

Recent advances in multi-modal large language models (MLLMs) have demonstrated strong performance across various domains; however, their ability to comprehend driving scenes remains less proven. The complexity of driving scenarios, which includes multi-view information, poses significant challenges for existing MLLMs. In this paper, we introduce NuPlanQA-Eval, a multi-view, multi-modal evaluation benchmark for driving scene understanding. To further support generalization to multi-view driving scenarios, we also propose NuPlanQA-1M, a large-scale dataset comprising 1M real-world visual question-answering (VQA) pairs. For context-aware analysis of traffic scenes, we categorize our dataset into nine subtasks across three core skills: Road Environment Perception, Spatial Relations Recognition, and Ego-Centric Reasoning. Furthermore, we present BEV-LLM, integrating Bird's-Eye-View (BEV) features from multi-view images into MLLMs. Our evaluation results reveal key challenges that existing MLLMs face in driving scene-specific perception and spatial reasoning from ego-centric perspectives. In contrast, BEV-LLM demonstrates remarkable adaptability to this domain, outperforming other models in six of the nine subtasks. These findings highlight how BEV integration enhances multi-view MLLMs while also identifying key areas that require further refinement for effective adaptation to driving scenes. To facilitate further research, we publicly release NuPlanQA at https://github.com/sungyeonparkk/NuPlanQA.

驾驶理解多模态大模型数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。