提出可量化危险性的框架,提前预警高风险决策。
Criticality and Safety Margins for Reinforcement Learning
- 用偏离策略后奖励下降量定义真实危险性
- 设计低开销代理指标,与真实危险性单调相关
- 安全边际直观易懂,适合部署时人工干预
当前强化学习方法常遭遇不安全状态。为在事后分析或部署中及时识别此类情况,需建立可靠的危险性评估体系。本文提出真实危险性:当智能体连续执行n次随机动作时,预期奖励的下降量。引入代理危险性作为低开销指标,与真实危险性具有统计单调关系。安全边际定义为在高置信度下性能损失不超过容忍值所能承受的随机动作数。实验显示,在A3C算法于Atari Beamrider环境中的表现中,最低5%的安全边际覆盖了47%的智能体损失;即仅监督5%的决策,可能防止近一半错误。该框架可在决策前量化潜在危害,提升自主系统调试与监管效率。
原文摘要 · Abstract (English)
State of the art reinforcement learning methods sometimes encounter unsafe situations. Identifying when these situations occur is of interest both for post-hoc analysis and during deployment, where it might be advantageous to call out to a human overseer for help. Efforts to gauge the criticality of different points in time have been developed, but their accuracy is not well established due to a lack of ground truth, and they are not designed to be easily interpretable by end users. Therefore, we seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality. Safety margins make these interpretable, when defined as the number of random actions for which performance loss will not exceed some tolerance with high confidence. We demonstrate this approach in several environment-agent combinations; for an A3C agent in an Atari Beamrider environment, the lowest 5% of safety margins contain 47% of agent losses; i.e., supervising only 5% of decisions could potentially prevent roughly half of an agent's errors. This criticality framework measures the potential impacts of bad decisions, even before those decisions are made, allowing for more effective debugging and oversight of autonomous agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。