用多个仿真器联合测试,找出通用失效场景,提升自动驾驶系统测试可信度。
Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing
- 通过多仿真器协同搜索,优先筛选跨平台一致失效的测试场景。
- 相比单仿真器,失败率提升66%;比现有方法多发现3.4倍通用失效测试。
- 用预测模型提前识别仿真器分歧,减少无效测试,提升效率。
基于场景的驾驶仿真测试广泛用于识别自动驾驶辅助系统(ADAS)的失效条件。然而,同一或不同仿真器重复测试可能产生不一致结果,源于仿真脆弱性或物理实现差异。本文提出MultiSim,一种基于搜索的多仿真测试方法,利用仿真器集成来识别诱发失效且与仿真器无关的测试场景。在搜索过程中,每个场景在多个仿真器上联合评估:跨仿真器结果一致的场景优先探索,仅部分仿真器失效的则低优先级处理,因其可能反映仿真器特异性问题。实验测试三个车道保持型ADAS在三组主流仿真器上的表现,结果显示,MultiSim平均比单仿真器测试提高66%的仿真器无关失效率;相较于现有先进多仿真方法(独立生成后合并),可平均多发现3.4倍的仿真器无关失败测试,并获得更高失败率。为避免在仿真器意见不一时执行昂贵测试,我们引入代理模型预测分歧并跳过测试。结果表明,该策略在保持有效失败数量的同时显著提升效率。研究证实,仿真器集成是实现自动驾驶测试自动跨复现的有前景路径。
原文摘要 · Abstract (English)
Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS). However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics. In this paper, we present MultiSim, a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS on different pairs of three widely used simulators, demonstrates that MultiSim outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators, MultiSim identifies, on average, up to 3.4X more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search retains the average number of valid failures and also improves efficiency. Our findings indicate that combining an ensemble of simulators is a promising approach for the automated cross-replication in ADAS testing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。