arXiv:2608.23906cs.AI2026-08

提出评估AI在复杂系统中风险的框架,量化其对金融等关键系统的潜在危害。

Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems

论文配图:Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems
图 1 · 摘自论文原文
  • 融合危险分析与概率建模,从模型行为推导系统级影响
  • 实验证明简单对抗输入可导致AI推荐被采纳并引发连锁崩溃
  • 适合关注AI治理、系统安全与金融风险的从业者

人工智能日益融入复杂社会技术系统,包括关键国家基础设施(CNI),其危害源于技术、人力与组织要素间的交互。然而当前的AI评估仍以模型为中心,难以揭示行为如何演变为系统级风险。本文提出一种框架,结合结构化危险分析、组件级测试与概率系统建模,建立从模型行为到系统结果的可追溯路径。该框架使从业者能够回答AI失败的‘所以怎样’问题,量化其系统性影响,并推动基于证据和前瞻性的AI治理。以英国实时全额结算(RTGS)系统为例,采用系统理论过程分析(STPA)推导出由AI驱动的损失情景,并考察大语言模型(LLM)交易受恶意操纵的情形。组件级实验显示,简单对抗输入即可引发可测量的行为改变,导致AI建议被采纳。在所用的金融传染模型中,这些变化削弱系统韧性,增加银行倒闭数量,并降低触发级联中断的冲击阈值,尤其在广泛或垄断式采用AI时更为显著。

原文摘要 · Abstract (English)

Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.

AI风险系统安全金融系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。