针对强化学习策略的高风险状态进行重点测试,提升安全验证效率。
Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learning
- 基于模型计算状态重要性排序,优先测试影响最大的状态。
- 仅用少量采样即能证明整个状态空间的安全性保证。
- 适合对安全性要求高的强化学习应用,如自动驾驶、机器人控制。
在许多深度强化学习(RL)任务中,策略在不同状态下的决策对预期安全性和性能具有不同影响。由于RL策略高度复杂,测试应聚焦于决策影响最大的状态。本文提出一种新型基于模型的方法,严格计算整个状态空间中各状态的重要性排名,并将测试重点集中于排名最高的状态。本研究聚焦安全测试,但方法可扩展至性能测试。每轮测试中,框架计算乐观与悲观的安全估计,提供政策执行在所有建模状态上的期望结果上下界。收敛后,方法将状态空间划分为安全与不安全区域,揭示策略弱点。该方法具备两大特性:(1) 最优测试用例选择——测试过程始终评估最关乎安全性的状态;(2) 安全性保障——通过采样部分策略即可对整个状态空间提供形式化验证保证,悲观估计确认的安全性质可被正式证明成立。我们在多个实例上进行了详细评估,结果表明,该方法以极低测试开销即可发现不安全策略行为。
原文摘要 · Abstract (English)
In many Deep Reinforcement Learning (RL) problems, decisions in a trained policy vary in significance for the expected safety and performance of the policy. Since RL policies are very complex, testing efforts should concentrate on states in which the agent's decisions have the highest impact on the expected outcome. In this paper, we propose a novel model-based method to rigorously compute a ranking of state importance across the entire state space. We then focus our testing efforts on the highest-ranked states. In this paper, we focus on testing for safety. However, the proposed methods can be easily adapted to test for performance. In each iteration, our testing framework computes optimistic and pessimistic safety estimates. These estimates provide lower and upper bounds on the expected outcomes of the policy execution across all modeled states in the state space. Our approach divides the state space into safe and unsafe regions upon convergence, providing clear insights into the policy's weaknesses. Two important properties characterize our approach. (1) Optimal Test-Case Selection: At any time in the testing process, our approach evaluates the policy in the states that are most critical for safety. (2) Guaranteed Safety: Our approach can provide formal verification guarantees over the entire state space by sampling only a fraction of the policy. Any safety properties assured by the pessimistic estimate are formally proven to hold for the policy. We provide a detailed evaluation of our framework on several examples, showing that our method discovers unsafe policy behavior with low testing effort.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。