用大模型生成更真实、多样的自动驾驶测试场景。
Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
- 利用大模型融合语言、图像、地图等多模态输入生成驾驶场景。
- 涵盖5类模型架构,支持复杂场景合成与分析。
- 适合自动驾驶安全验证与仿真测试研究人员。
自动驾驶车辆在复杂环境中的安全导航依赖于对多样化且罕见驾驶场景的处理。基于模拟和场景的测试已成为开发与验证自动驾驶系统的关键方法。传统场景生成依赖规则系统、知识驱动模型和数据驱动合成,常导致多样性不足且难以生成真实的高风险案例。随着基础模型(foundation models)的出现——这类预训练的通用人工智能模型可处理自然语言、传感器数据、高精地图及控制动作等异构输入——为复杂驾驶场景的合成与解析提供了新途径。本文系统综述了截至2025年5月,基础模型在自动驾驶场景生成与分析中的应用。提出统一分类体系,涵盖大语言模型、视觉-语言模型、多模态大语言模型、扩散模型与世界模型。同时梳理了相关方法、开源数据集、仿真平台、基准挑战及专用于场景生成与分析的评估指标。最后指出当前开放挑战与未来研究方向。所有被评论文均收录于持续更新的资源库中,链接为 https://github.com/TUM-AVS/FM-for-Scenario-Generation-Analysis。
原文摘要 · Abstract (English)
For autonomous vehicles, safe navigation in complex environments depends on handling a broad range of diverse and rare driving scenarios. Simulation- and scenario-based testing have emerged as key approaches to development and validation of autonomous driving systems. Traditional scenario generation relies on rule-based systems, knowledge-driven models, and data-driven synthesis, often producing limited diversity and unrealistic safety-critical cases. With the emergence of foundation models, which represent a new generation of pre-trained, general-purpose AI models, developers can process heterogeneous inputs (e.g., natural language, sensor data, HD maps, and control actions), enabling the synthesis and interpretation of complex driving scenarios. In this paper, we conduct a survey about the application of foundation models for scenario generation and scenario analysis in autonomous driving (as of May 2025). Our survey presents a unified taxonomy that includes large language models, vision-language models, multimodal large language models, diffusion models, and world models for the generation and analysis of autonomous driving scenarios. In addition, we review the methodologies, open-source datasets, simulation platforms, and benchmark challenges, and we examine the evaluation metrics tailored explicitly to scenario generation and analysis. Finally, the survey concludes by highlighting the open challenges and research questions, and outlining promising future research directions. All reviewed papers are listed in a continuously maintained repository, which contains supplementary materials and is available at https://github.com/TUM-AVS/FM-for-Scenario-Generation-Analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。