构建首个城市驾驶场景时空推理数据集,提升自动驾驶视觉理解能力
STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes
- 基于东京100小时多传感器数据,生成27万帧含3D标注的问答对
- 新模型在空间定位上达55%准确率,未来运动预测一致性达28%
- 适合自动驾驶、多模态大模型、时空推理方向研究者使用
视觉语言模型(VLMs)已用于自动驾驶决策支持,但其基于静态网络图像-文本对的训练,难以满足动态交通场景所需的精确时空推理。为此,我们提出STRIDE-QA,一个大规模面向城市驾驶场景的视觉问答(VQA)数据集,从东京100小时多传感器驾驶数据中构建,覆盖复杂多样路况。该数据集包含27万帧图像上的1600万组问答对,配有密集自动标注的3D边界框、分割掩码和多目标轨迹。通过三个新颖的问答任务,支持对象中心与自车视角的时空推理。基准测试显示,现有VLMs在预测一致性上几乎为零;而经STRIDE-QA微调后的模型,空间定位成功率提升至55%,未来运动预测一致性达28%,显著优于通用VLMs。STRIDE-QA为开发更可靠的自动驾驶安全系统提供了坚实基础。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have been applied to autonomous driving to support decision-making in complex real-world scenarios. However, their training on static, web-sourced image-text pairs fundamentally limits the precise spatiotemporal reasoning required to understand and predict dynamic traffic scenes. We address this critical gap with STRIDE-QA, a large-scale visual question answering (VQA) dataset for physically grounded reasoning from an ego-centric perspective. Constructed from 100 hours of multi-sensor driving data in Tokyo, capturing diverse and challenging conditions, STRIDE-QA is the largest VQA dataset for spatiotemporal reasoning in urban driving, offering 16M QA pairs over 270K frames. Grounded by dense, automatically generated annotations including 3D bounding boxes, segmentation masks, and multi-object tracks, the dataset uniquely supports both object-centric and ego-centric reasoning through three novel QA tasks that require spatial localization and temporal prediction. Our benchmarks demonstrate that existing VLMs struggle significantly, with near-zero scores on prediction consistency. In contrast, VLMs fine-tuned on STRIDE-QA exhibit dramatic performance gains, achieving 55% success in spatial localization and 28% consistency in future motion prediction, compared to near-zero scores from general-purpose VLMs. Therefore, STRIDE-QA establishes a comprehensive foundation for developing more reliable VLMs for safety-critical autonomous systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。