arXiv:2603.25197cs.AIcs.ET2026-03

AI助安全分析可能隐藏盲区,关键在协作流程设计。

The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

  • 构建五维能力框架,量化安全工程中的专业能力
  • 提出‘能力阴影’概念,揭示AI导致人类思维被系统性压缩
  • 证明阴影效应乘积放大,需以流程而非工具评估可信度

随着AI助手逐步融入物理AI系统安全工程工作流,一个核心问题浮现:AI辅助是否真正提升安全分析质量,还是引入仅在部署后才显现的系统性盲点?本文建立安全分析中AI协助的形式化框架。首先阐明安全工程难以通过基准评估的原因:安全能力本质上多维、受上下文约束、存在固有不完整性且专家意见合理分歧。我们通过包含领域知识、标准专长、操作经验、情境理解与判断力的五维能力框架进行形式化。提出‘能力阴影’概念——即AI生成的安全分析所引发的人类推理范围系统性缩小,阴影并非AI输出的内容,而是它阻止被考虑的部分。我们形式化四种典型人-AI协作结构,并推导出闭式性能边界,证明能力阴影呈乘积式累积,导致退化程度远超简单加法估计。核心结论是:安全工程中的AI辅助本质是协作流程设计问题,而非软件采购决策。同一工具的使用方式决定其是提升还是削弱分析质量。我们推导出抗阴影的工作流非退化条件,呼吁从工具资格转向流程资格,以实现可信的物理AI。

原文摘要 · Abstract (English)

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.

安全工程人机协作能力阴影物理AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。