用大模型辅助提前发现AI系统隐患,降低事故风险。
From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems
- 用LLM改造STPA流程,半自动识别ML系统潜在风险
- 实验证明可预见大量未被察觉的隐患问题
- 适合缺乏安全专家的开发团队快速落地安全设计
机器学习组件日益嵌入软件产品,但其复杂性和内在不确定性常引发意外且危险的后果。尽管存在风险,实践者很少采用前瞻性方法预防灾害。传统安全工程方法如故障模式影响分析(FMEA)和系统理论过程分析(STPA)虽提供系统化早期风险识别框架,却鲜被采纳。本文主张将危害分析融入所有ML驱动软件的开发流程,并呼吁增强支持以提升开发者可及性。通过使用大语言模型(LLMs)部分自动化经修改的STPA流程,并在关键步骤保留人工监督,我们期望解决两大挑战:对资深安全专家的高度依赖,以及传统危害分析耗时费力,常阻碍其融入实际开发流程。文中以一个持续案例说明,许多看似不可预见的问题实际上可被提前识别。
原文摘要 · Abstract (English)
Machine learning (ML) components are increasingly integrated into software products, yet their complexity and inherent uncertainty often lead to unintended and hazardous consequences, both for individuals and society at large. Despite these risks, practitioners seldom adopt proactive approaches to anticipate and mitigate hazards before they occur. Traditional safety engineering approaches, such as Failure Mode and Effects Analysis (FMEA) and System Theoretic Process Analysis (STPA), offer systematic frameworks for early risk identification but are rarely adopted. This position paper advocates for integrating hazard analysis into the development of any ML-powered software product and calls for greater support to make this process accessible to developers. By using large language models (LLMs) to partially automate a modified STPA process with human oversight at critical steps, we expect to address two key challenges: the heavy dependency on highly experienced safety engineering experts, and the time-consuming, labor-intensive nature of traditional hazard analysis, which often impedes its integration into real-world development workflows. We illustrate our approach with a running example, demonstrating that many seemingly unanticipated issues can, in fact, be anticipated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。