为智能体系统团队设计风险治理框架,解决传统软件管理失效问题。
Risk Architecture for AI-Native Engineering Teams: An Organizational Framework for Agentic System Governance

- 提出七维团队分类与六类故障模式,识别智能体系统的独特风险。
- 发现越接近纯AI原生团队,风险覆盖越差,高危故障在边界处突增。
- 适合工程管理者、AI系统架构师及组织级风险决策者阅读。
工程管理研究已有成熟的软件风险框架:按功能划分责任、按严重性升级、通过测试覆盖率保障。这些框架隐含了确定性行为、离散可审计变更、清晰的组件-责任人映射假设。但构建与运营智能体AI系统的团队同时违背这三个假设:输出具有概率性,系统自主执行多步操作,风险面在部署间悄然演变。现有AI风险文献或从上层(如NIST AI RMF、ISO/IEC 42001)或下层(如OWASP智能体安全指南)切入,却未触及工程经理(EM)实际操作层——角色权责与升级机制。本文贡献:(i) 七个维度的团队画像,区分纯软件工程、混合与AI原生团队;(ii) 六类故障模式分类,包含此前未明确的‘依赖边界确定性不匹配’类别;(iii) 一套合成框架适配度评估方法,衡量各团队画像在检测、控制和升级特定场景时的表现。因研究对象是框架适配度而非人类行为,评估结果为推导而非观测覆盖。随着团队从纯软件工程向AI原生演进,覆盖度单调下降,中位数降低,未覆盖的高后果故障数量骤增,且最严重、最不被覆盖的故障并非出自AI原生团队本身,而是其概率输出被假定确定性的依赖方所消耗的组织边界处。
原文摘要 · Abstract (English)
Engineering management research has produced mature frameworks for software risk: ownership by feature, escalation by severity, and assurance by test coverage. These frameworks implicitly assume deterministic behavior, discrete and auditable change events, and clear component-to-owner mappings. Teams that build and operate agentic AI systems violate all three assumptions at once: outputs are probabilistic, systems take autonomous multi-step actions, and the risk surface mutates silently between deployments. Existing AI risk literature addresses this from above (policy frameworks such as the NIST AI RMF and ISO/IEC 42001) or below (threat taxonomies such as OWASP's agentic AI guidance), but not at the layer where an engineering manager (EM) operates: roles, decision rights, and escalation structures. This paper contributes (i) a seven-dimension profile distinguishing pure software-engineering, hybrid, and AI-native teams; (ii) a six-cluster failure-mode taxonomy including a previously unarticulated cluster, dependency-boundary determinism mismatch; and (iii) a synthetic framework-adequacy methodology scoring how well each profile's risk architecture detects, contains, and escalates a defined scenario set. Because the object of study is framework adequacy rather than human behavior, the evaluation yields derived rather than observed coverage claims. Coverage degrades as teams move from pure software engineering to AI-native operation, monotonically in the median and abruptly in the count of uncovered, high-consequence failures appearing only at the AI-native step. The degradation concentrates in specific failure-mode categories, and the most severe, least-covered failures arise not inside AI-native teams but at the organizational boundary where their probabilistic outputs are consumed by determinism-assuming dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。