从重大事故中汲取教训，让AI设计更重视组织与社会因素。
Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
- 从切尔诺贝利等灾难看，风险常被忽视非因技术不可控，而是组织问题。
- 强调组织层面的风险感知、沟通与责任追溯，是避免AI系统失控的关键。
- 适合关注AI伦理、系统安全与组织管理的研究者与开发者阅读。
随着自动化决策和数据驱动技术广泛应用于社会关键领域，理解其在具体情境中的能力、局限与风险，需对完整社会技术系统进行分析。对高度复杂系统中风险的社会技术分析，为人工智能系统的构建与评估提供了明确启示：应超越仅关注可靠或‘负责任’组件的技术视角，转而从系统层面理解风险。人类制造的灾难已持续研究数十年，如切尔诺贝利、三里岛、福岛第一核电站、博帕尔、挑战者号航天飞机事故。普遍误解是这些事件属于难以预见的偶然事故，源于复杂系统中不可测的交互。深入分析表明，这些风险和隐患事前早已明确，却因社会结构、政治与经济因素未被采取行动。本文提出，人工智能的发展与应用可从这些未吸取的教训中获益：提升组织层面的风险感知、沟通与分析能力；确保需求与责任的可追溯性；建立包含社会与组织动态的第一性工程考量的综合责任与安全体系。针对每一领域，本文列出具体未吸取的教训，并以历史事故为例说明其失败原因，同时指出现代计算系统（尤其是人工智能）仍存在类似盲区。
原文摘要 · Abstract (English)
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or "responsibly designed" components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。