揭示AI系统中的安全与安全技术债务,提出可落地的管理框架。
On AI Safety and Security Technical Debt in Engineering AI-Enabled Systems

- 从信任维度重构技术债务,识别31类AI特有债务
- 梳理60篇研究,提炼8条安全、26条安全缓解指南
- 构建AITD-MAP框架,助力工程师识别与化解风险
人工智能系统在医疗、自动驾驶、金融和教育等高风险领域日益普及。尽管具备强大的数据驱动与自适应能力,其复杂性、快速迭代及对动态数据管道的依赖,带来了新型工程责任,统称为人工智能技术债务(AITD)。AITD源于数据治理、模型实现、算法设计、架构决策、运营流程、文档实践和测试充分性等多个层面。不同于传统技术债务,许多AITD具有隐蔽性,并在高度耦合的AI流水线中传播,导致维护困难、可靠性下降以及安全或隐私风险上升。本文基于人工智能可信、风险与安全管理(AI TRiSM)原则,重新审视技术债务,聚焦于安全与安全方面的技术债务。通过系统综述60项原始研究,识别出31种不同类型的AITD,构建以根因为导向的七类分类体系。分析显示这些债务映射至18项信任相关关切,包括6项安全危害和12项安全漏洞。为支持缓解,研究整合了34条可操作建议(8条安全、26条安全),覆盖整个AI生命周期的预防、检测与减缓。在此基础上,提出AITD-MAP框架,将分类体系、质量与风险影响及缓解策略统一整合,形成风险感知的AI工程结构。该框架旨在帮助AI软件工程师显式化安全与安全技术债务,理解其根本原因,并采取有效措施减轻其影响。
原文摘要 · Abstract (English)
Artificial intelligence (AI) systems are increasingly deployed in high-stakes domains such as healthcare, autonomous driving, finance, and education. While these systems offer powerful data-driven and adaptive capabilities, their complexity, rapid evolution, and dependence on dynamic data pipelines introduce new forms of engineering liability collectively referred to as AI Technical Debts (AITDs). AITDs arise from root causes spanning data governance, model implementation, algorithm design, architectural decisions, operational processes, documentation practices, and testing adequacy. Unlike conventional technical debt, many AITDs are latent and propagate across tightly coupled AI pipelines, leading to maintenance challenges, reliability degradation, and heightened safety or security risks. Guided by the principles of AI Trust, Risk, and Security Management (AI TRiSM), this study reinterprets technical debt through the interconnected dimensions of trustworthiness, focusing on AI safety and security technical debts. We conduct a systematic review of 60 primary studies and identify 31 distinct types of AITD, which are organized into a root-cause-oriented taxonomy comprising seven classes. The analysis examines how these debts map to 18 trust-related concerns, including 6 safety hazards and 12 security vulnerabilities. To support mitigation, the review synthesizes 34 actionable guidelines (8 safety and 26 security) targeting the prevention, detection, and reduction of AITDs across the AI lifecycle. Building on these findings, we introduce AITD-MAP, an integrated framework that connects the AITD taxonomy, quality and risk impacts, and mitigation strategies into a unified structure for risk-aware AI engineering. The framework aims to assist AI software engineers in making AI safety and security technical debts visible, understanding their root causes, and mitigating their presence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。