将真实驾驶视频自动转为可重复模拟的交通场景
From Driving Videos to Simulatable Scenarios
- 用视觉语言模型提取视频语义，再由大模型生成可执行场景
- 生成场景中90%的关键元素与原视频一致
- 适合自动驾驶安全评估与仿真系统开发者
自动驾驶车辆面临从日常交通到罕见事件的各种驾驶场景。为评估安全性，必须以可控、可重复且可扩展的方式重现这些场景，而仿真在此过程中起关键作用。本文提出 D-V2S 框架，可从驾驶视频自动生成可模拟的交通场景。该框架分两阶段运行：首先，驾驶记录分析器（DRA）利用视觉语言模型（VLM）和设计的提示词，从输入视频生成自然语言描述，捕捉道路布局与动态交通交互；随后，场景生成器（SG）使用大语言模型（LLM）和我们的条件上下文，将这些描述转化为可执行场景。通过仿真验证，D-V2S 生成的场景中，90% 的相关语义元素与原始视频一致。我们还提供了定性结果，展示 D-V2S 将真实驾驶视频转换为可模拟场景的能力。此外，我们进行了语义与人工驱动的消融分析，证明 VLM 选择对 DRA 的影响，并显示我们的 SG 在人类偏好测试中达到 75% 的胜率，优于其他先进方法。
原文摘要 · Abstract (English)
Autonomous vehicles (AVs) face driving scenarios ranging from routine traffic to rare events. To assess safety it is crucial to reproduce these scenarios in a controllable, repeatable, and scalable manner, with simulation playing a key role. This paper introduces D-V2S, a novel framework that automatically generates simulatable driving scenarios from driving videos. D-V2S operates in two stages: a Driving Record Analyzer (DRA) uses a vision language model (VLM) with our designed prompt to produce natural-language descriptions from input videos, capturing road layouts and dynamic traffic interactions; subsequently, a Scenario Generator (SG) uses a large language model (LLM) and our conditioning context to translate these descriptions into executable scenarios. Using simulations, we show that D-V2S generates scenarios where 90% of the relevant semantic elements of the videos are present. We also provide qualitative results demonstrating D-V2S's capability to transform real-world driving videos into simulatable scenarios. Moreover, we provide both semantic and human driven ablative analyses of D-V2S's modules. In particular, we show how the VLM choice matters for DRA, and how our SG achieves a 75% preference rate over other state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。