AI发展中的组织性风险被忽视,可能引发灾难性事故。
The Normalization of Deviance in AI Development
- 用航天、核能、航空事故类比,揭示AI组织的系统性失效机制
- 合规流程无法防止重大事故,安全体系存在结构性漏洞
- 适合关注AI治理与组织安全的从业者和研究者阅读
关于人工智能风险的研究主要聚焦于能力风险:即系统过于强大、自主或与人类价值观错位。较少关注的是组织层面——构建这些系统的机构是否本身容易滑向失败。本文认为确实如此。无论人工智能系统能力如何,其背后组织都面临与过去重大技术灾难相同的结构性动因。通过分析挑战者号航天飞机、三里岛核事故以及波音737 MAX坠毁案例,本文识别出每次灾难前共有的结构性机制,并将其映射到当前的AI开发中。研究发现,现有安全基础设施可能比表面上看起来更脆弱,因为组织可在完全合规的情况下仍导致灾难性后果。目前,AI发展的灾前阶段仍在进行中;本文目的正是在可干预时将这些动态显性化。
原文摘要 · Abstract (English)
Work on the risks of artificial intelligence has focused predominantly on capability risk: the danger that systems become too powerful, too autonomous, or too misaligned with human values. Far less attention has been paid to the organizational level---to whether the institutions building these systems are themselves predisposed to drift toward failure. This paper argues that they are. Regardless of how capable AI systems become, the organizations building them face the same structural dynamics that preceded past major technological disasters. Drawing on case studies of the Space Shuttle Challenger, the Three Mile Island accident, and the Boeing 737 MAX crashes, this paper identifies the common structural mechanisms preceding each failure and maps them onto contemporary AI development. The findings suggest that existing safety infrastructure may provide less protection than it appears, as organizations can complete safety processes in full compliance and still produce catastrophic outcomes. The pre-disaster period of AI development is still underway; the purpose of this paper is to make these dynamics legible while they can still be interrupted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。