用真实交通场景构建VQA数据集，提升视觉语言模型的环境理解能力。
Embodied Scene Understanding for Vision Language Models via MetaVQA
- 基于nuScenes和Waymo数据生成带空间标注的问答对。
- 微调后模型在模拟中推理准确率提升，出现安全驾驶行为。
- 成果可从仿真迁移到真实场景，适合自动驾驶研究者。
视觉语言模型（VLMs）在移动类应用中展现出作为具身智能体的巨大潜力。然而，缺乏标准化的闭环评估基准来检验其空间推理与序列决策能力。为此，我们提出MetaVQA：一个通过视觉问答（VQA）与闭环模拟评估并提升VLM空间关系与场景动态理解能力的综合性基准。MetaVQA利用Set-of-Mark提示与nuScenes、Waymo数据集的俯视图真值标注，自动生成基于多样化真实交通场景的海量问答对，确保指令以物体为中心且上下文丰富。实验表明，使用MetaVQA数据集微调VLM能显著提升其在安全关键型模拟中的空间推理与具身场景理解能力，不仅体现在VQA准确率提升，还表现为涌现的安全驾驶行为。此外，该学习具有强泛化能力，可从仿真迁移至真实观测。代码与数据将公开于https://metadriverse.github.io/metavqa。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) demonstrate significant potential as embodied AI agents for various mobility applications. However, a standardized, closed-loop benchmark for evaluating their spatial reasoning and sequential decision-making capabilities is lacking. To address this, we present MetaVQA: a comprehensive benchmark designed to assess and enhance VLMs' understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations. MetaVQA leverages Set-of-Mark prompting and top-down view ground-truth annotations from nuScenes and Waymo datasets to automatically generate extensive question-answer pairs based on diverse real-world traffic scenarios, ensuring object-centric and context-rich instructions. Our experiments show that fine-tuning VLMs with the MetaVQA dataset significantly improves their spatial reasoning and embodied scene comprehension in safety-critical simulations, evident not only in improved VQA accuracies but also in emerging safety-aware driving maneuvers. In addition, the learning demonstrates strong transferability from simulation to real-world observation. Code and data will be publicly available at https://metadriverse.github.io/metavqa .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。