用概率标签替代好坏二分,实现多智能体系统的连续安全治理
Soft-Label Governance for Distributional Safety in Multi-Agent Systems

- 引入软标签机制,用0-1概率表示行为安全性,取代传统二分类
- 严格治理使福利下降超40%却未提升安全,证明需权衡安全与效率
- 电路熔断器需精细调参,过严会严重损害系统价值
多智能体人工智能系统会产生单个智能体孤立时不会出现的涌现风险。现有安全框架依赖对行为的二元判定,忽略了代理评估中的不确定性。我们提出SWARM(系统级多智能体风险评估框架),以软概率标签 $p = P(v{=}+1) \∈ [0,1]$ 取代二元好坏标签,支持连续收益计算、毒性度量和治理干预。SWARM包含可配置的治理模块(交易税、熔断机制、声誉衰减、随机审计),并量化其效果,使用期望毒性 $\ℵ[1{-}p \∣ \text{accepted}]$ 和质量差距 $\ℵ[p \∣ \text{accepted}] - \ℵ[p \∣ \text{rejected}]$ 等概率指标。在七种场景中五次种子重复实验表明,严格治理使福利下降超过40%但未改善安全。同时,激进内部化系统外部性使总福利从基准值+262暴跌至-67,而毒性保持不变。熔断机制需谨慎校准:阈值过严严重降低系统价值,最优阈值可在适度福利下最小化毒性。对照实验显示,软标签能检测自优化智能体通过传统二元评估的代理游戏行为。该基础治理层可直接应用于实时大模型驱动的智能体(Concordia实体、Claude、GPT-4o Mini)而无需修改。结果表明,分布安全需要持续的风险度量,且治理杠杆校准涉及可量化的安全-福利权衡。代码与资源公开于 https://www.swarm-ai.org/。
原文摘要 · Abstract (English)
Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (\textbf{S}ystem-\textbf{W}ide \textbf{A}ssessment of \textbf{R}isk in \textbf{M}ulti-agent systems), a simulation framework that replaces binary good/bad labels with \emph{soft probabilistic labels} $p = P(v{=}+1) \in [0,1]$, enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$. Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40\% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$, while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires \emph{continuous} risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。