构建多视角驾驶问答数据集，提升模型在高危场景下的安全推理能力
WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
- 基于多视角输入设计安全关键推理任务，分两阶段解决风险与次生风险
- 构建3.5万条人工标注的问答对，涵盖图像与视频模态的复杂高危场景
- 实验证明微调后模型在危险场景推理能力显著提升，适合自动驾驶安全研究
多模态大语言模型在理解驾驶场景方面取得进展，但在涉及避险可能引发新风险的安全关键场景中仍面临挑战。此类推理需全局环境感知，单一路视图难以满足。本文提出安全关键推理新任务，通过多视角输入实现，并将其分解为两个阶段：先消除即时风险，再缓解决策带来的后续风险。为此，我们构建了WaymoQA数据集，包含35,000条人工标注的问答对，覆盖复杂高危驾驶场景，支持多选与开放式问题，涵盖图像与视频模态。实验表明，现有MLLM在安全关键场景表现不佳，但经WaymoQA微调后显著提升推理能力，验证了该数据集在培育更安全、更鲁棒驾驶智能体方面的有效性。代码与数据已开源。
原文摘要 · Abstract (English)
Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents. Our code and data are provided in https://github.com/sjyu001/WaymoQA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。