当智能体有策略协作时,即使信息泄露量小也可能造成重大危害。
A Note on the Strategic Confinement Problem
- 考虑具有协作能力的智能体,其通信可聚焦于关键敏感数据
- 低容量通道仍可能引发严重损害,因能精准选择高破坏性结果
- 适合研究安全系统、智能体博弈与隐蔽通信的学者参考
Lampson 的隔离问题关注如何防止处理机密信息的程序向第三方泄露信息。本文提出战略隔离问题,即当通信双方为具有协同资源的战略代理时,残余通信能力可集中于机密数据的低熵、高影响谓词上。因此,信息泄露的上限未必对应最大危害的上限:一个容量极小的信道仍可能足以选择破坏性结果。我们指出,由学习型战略代理构成的系统自然呈现此问题,因其行为规范无法完全定义,习得的沟通惯例对外部观察者不可预测或复现,且足够强的代理可构建难以检测或清除的隐秘通信方案。本文贡献并非新通信理论,而是对战略代理存在下的隔离问题的重新诠释:传统隔离仅限制信息流动,而战略隔离强调这并不限制代理协同所能达成的后果。
原文摘要 · Abstract (English)
Lampson's confinement problem asks how to prevent a program that processes confidential information from leaking it to a third party. We introduce the strategic confinement problem, which arises when the communicating parties are strategic agents with shared coordination resources. In this setting, residual communication capacity can be concentrated on low-entropy, high-impact predicates of the confidential data. Consequently, bounds on information leakage need not induce corresponding bounds on worst-case harm: a channel with negligible capacity may still suffice to select damaging outcomes. We argue that systems of learnt strategic agents naturally instantiate this problem because they do not admit complete behavioural specifications, their learnt conventions generally cannot be predicted or reproduced by an external observer, and sufficiently capable agents can construct covert communication schemes that are difficult to detect or eliminate. Our contribution is therefore not a new theory of communication, but a reinterpretation of confinement in the presence of strategic agents. Classical confinement bounds what information may flow; strategic confinement highlights that this need not bound what strategic agents can jointly achieve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。