arXiv:2602.18986cs.AI2026-02被引 1

提出贝叶斯框架量化高自动化系统故障传播风险,指导安全部署。

Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight

  • 用贝叶斯分解建模故障损失,分离出故障传播概率这一核心风险项。
  • 揭示自动化程度越高,故障演变为严重损失的概率越大,2012年骑士资本事件印证此规律。
  • 适用于金融、医疗等关键领域的自动化系统治理,适合决策者与风控工程师使用。

金融、医疗、交通、内容审核及关键基础设施等领域正快速部署高度自动化的AI系统,但缺乏量化自动化程度提升如何放大故障后果的系统方法。本文提出一个简洁的贝叶斯风险分解模型,将预期损失表示为三个因子的乘积:系统故障概率、给定自动化水平下故障演变为危害的条件概率,以及危害的预期严重性。该框架聚焦关键量——故障传播至危害的条件概率,涵盖执行与监督风险,而不仅限于模型准确率。我们建立了完整的理论基础:形式化证明了分解公式,提出危害传播等价定理,关联传播概率与可观测的执行控制机制,定义风险弹性度量,开展自动化政策的效率前沿分析,并给出资源最优分配原则(含二阶条件)。通过2012年骑士资本事件(4.4亿美元损失)作为典型案例,展示该框架对普遍失败模式的适用性,并阐明跨领域大规模实证验证所需的研究设计。本工作为智能体与自动化系统的部署导向型风险治理工具提供了理论基础。

原文摘要 · Abstract (English)

Organizations across finance, healthcare, transportation, content moderation, and critical infrastructure are rapidly deploying highly automated AI systems, yet they lack principled methods to quantify how increasing automation amplifies harm when failures occur. We propose a parsimonious Bayesian risk decomposition expressing expected loss as the product of three terms: the probability of system failure, the conditional probability that a failure propagates into harm given the automation level, and the expected severity of harm. This framework isolates a critical quantity -- the conditional probability that failures propagate into harm -- which captures execution and oversight risk rather than model accuracy alone. We develop complete theoretical foundations: formal proofs of the decomposition, a harm propagation equivalence theorem linking the harm propagation probability to observable execution controls, risk elasticity measures, efficient frontier analysis for automation policy, and optimal resource allocation principles with second-order conditions. We motivate the framework with an illustrative case study of the 2012 Knight Capital incident ($440M loss) as one instantiation of a broadly applicable failure pattern, and characterize the research design required to empirically validate the framework at scale across deployment domains. This work provides the theoretical foundations for a new class of deployment-focused risk governance tools for agentic and automated AI systems.

风险量化自动化系统贝叶斯框架治理工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。