用模拟的高难场景提升自动驾驶真实表现
RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case
- 构建13类高风险场景的仿真数据集HASS，覆盖昼夜雨晴等环境
- 通过提示工程与图像编码器，让大模型学会应对真实世界差异
- 在nuScenes上使复杂场景表现提升约50%，适合做自动驾驶安全优化
收集罕见高风险场景、长尾驾驶事件和复杂交互的真实世界数据仍具挑战性，导致现有自动驾驶系统在这些关键情形下表现不佳。本文提出RoboTron-Sim，通过模拟高难场景提升真实世界驾驶表现。首先，构建涵盖13类高风险边缘案例的仿真数据集HASS，包含昼夜、晴雨等平衡环境条件。其次，引入场景感知提示工程（SPE）和图像到本体编码器（I2E Encoder），使多模态大语言模型能有效从HASS中学习真实世界复杂驾驶技能，适应真实与仿真间的环境偏差及硬件差异。在nuScenes上的大量实验表明，RoboTron-Sim使复杂场景下的驾驶性能提升约50%，达到实时开环规划的顶尖水平。定性结果进一步验证其在处理罕见高风险场景中的有效性。
原文摘要 · Abstract (English)
Collecting real-world data for rare high-risk scenarios, long-tailed driving events, and complex interactions remains challenging, leading to poor performance of existing autonomous driving systems in these critical situations. In this paper, we propose RoboTron-Sim that improves real-world driving in critical situations by utilizing simulated hard cases. First, we develop a simulated dataset called Hard-case Augmented Synthetic Scenarios (HASS), which covers 13 high-risk edge-case categories, as well as balanced environmental conditions such as day/night and sunny/rainy. Second, we introduce Scenario-aware Prompt Engineering (SPE) and an Image-to-Ego Encoder (I2E Encoder) to enable multimodal large language models to effectively learn real-world challenging driving skills from HASS, via adapting to environmental deviations and hardware differences between real-world and simulated scenarios. Extensive experiments on nuScenes show that RoboTron-Sim improves driving performance in challenging scenarios by around 50%, achieving state-of-the-art results in real-world open-loop planning. Qualitative results further demonstrate the effectiveness of RoboTron-Sim in better managing rare high-risk driving scenarios. Project page: https://stars79689.github.io/RoboTron-Sim/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。