构建可验证的驾驶场景因果推理数据集,助力自动驾驶泛化能力评估
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving

- 提出时空因果结构化表示,显式记录驾驶中各主体的动作与依赖关系
- 构建2066段人类标注的驾驶片段数据集,含超3.4万条时空因果要素
- 适用于评估模型推理能力及验证自动生成标签质量,适合自动驾驶研究者
推理是实现自动驾驶在长尾场景下泛化的重要途径,因为它能推断场景元素间的依赖关系并据此得出观测之外的结论。然而,现有方法难以验证模型推理是否符合真实依赖:文本推理缺乏时空定位,时空图谱缺少因果连接,而大规模标注多由模型生成且难验证。为此,本文提出CASCADE(驾驶环境的时空因果分析),包含两个部分:(1) 驾驶场景的结构化推理表示;(2) 基于此构建的人类标注数据集。对每个与自身车辆互动的参与者,CASCADE表示在可见期间逐帧记录其动作、位置及其对其他主体行为和状态的依赖。该结构使推理预测可逐项机器验证,无需依赖(大)语言模型评判。CASCADE数据集涵盖PhysicalAI数据集中2,066段驾驶视频,共包含超过34,000个要素,包括8.6K条时间戳化的自身及智能体动作、3.7K条因果链接、2.9K条潜在影响,以及6.1K条关于智能体、物体、交通灯和环境的标注。作为全人工标注基准,可用于对比物理智能模型的推理能力,并验证自动生成推理标签的质量。数据集已公开于 https://huggingface.co/datasets/nvidia/cascade。
原文摘要 · Abstract (English)
Reasoning is a promising route to the generalization that autonomous driving requires in the long tail, as it can infer how the elements of a scene depend on one another and traverse those dependencies to conclusions beyond what is observed. Yet it is hard to tell whether a model's conclusions follow the scene's dependencies, because no driving representation makes them explicit enough to test against. Text-based reasoning traces lack spatio-temporal grounding, spatio-temporal scene graphs lack causal links, and reasoning annotations at scale are increasingly model-generated and hard to verify. To this end, we introduce CASCADE (Causal Spatio-Temporal Analysis of Driving Environments), which encompasses two components: (1) a structured scene representation for reasoning in driving scenes and (2) a human-annotated dataset built on it. For every actor that interacts with the ego vehicle, the CASCADE representation records frame-by-frame, for as long as the actor is visible, what action is taken, where it occurs, and how it depends on the actions and states of others. The resulting structure makes reasoning predictions machine-verifiable: they can be scored against it element by element, without relying on (M)LLM judges. The CASCADE dataset provides comprehensive human annotations for 2,066 driving clips of the PhysicalAI dataset, with over 34K elements that establish the spatio-temporal and causal context of each scene, including 8.6K time-stamped ego and agent actions, 3.7K causal links and 2.9K potential influences, and 6.1K annotations for agents, objects, traffic lights, and environments. Being entirely human-annotated, CASCADE provides the reference for this comparison: benchmarking the reasoning abilities of Physical AI models, and verifying the quality of automatically generated reasoning labels. The CASCADE dataset is available at https://huggingface.co/datasets/nvidia/cascade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。