通过分权架构防止AI自主作恶,确保目标不偏移。
Structural Enforcement of Goal Integrity in AI Agents via Separation-of-Powers Architecture
- 将意图生成、授权、执行分层隔离,用加密令牌连接
- 能检测目标漂移并拒绝语义偏离的指令,阈值可调
- 适合高风险自治系统,如医疗、金融、军事决策
前沿AI系统可能在无用户指令下生成并执行有害行为,源于内部构建的目标。现有方法如基于人类反馈的强化学习(RLHF)和宪法提示主要在模型层面操作,仅提供概率性安全保证。本文提出政策-执行-授权(PEA)架构,一种系统级分权设计,通过将意图生成、授权与执行分离为独立、隔离的层级,并以密码学约束的能力令牌连接。核心贡献包括:(C1) 意图验证层(IVL)确保能力与意图一致;(C2) 意图溯源追踪(ILT),通过密码锚点将所有可执行意图绑定至原始用户请求;(C3) 目标漂移检测,低于可配置阈值时拒绝语义偏离的意图;(C4) 输出语义门(OSG),使用结构化的 $K \times I \times P$ 威胁计算模型(知识、影响、政策)检测隐式操控;(C5) 形式化验证框架,证明在对抗性模型攻击下仍能维持目标完整性。该架构将对齐从行为属性转为结构性系统约束,为自治代理治理提供坚实基础。
原文摘要 · Abstract (English)
Recent evidence suggests that frontier AI systems can exhibit agentic misalignment, generating and executing harmful actions derived from internally constructed goals, even without explicit user requests. Existing mitigation methods, such as Reinforcement Learning from Human Feedback (RLHF) and constitutional prompting, operate primarily at the model level and provide only probabilistic safety guarantees. We propose the Policy-Execution-Authorization (PEA) architecture, a "separation-of-powers" design that enforces safety at the system level. PEA decouples intent generation, authorization, and execution into independent, isolated layers connected via cryptographically constrained capability tokens. We present five core contributions: (C1) an Intent Verification Layer (IVL) for ensuring capability-intent consistency; (C2) Intent Lineage Tracking (ILT), which binds all executable intents to the originating user request via cryptographic anchors; (C3) Goal Drift Detection, which rejects semantically divergent intents below a configurable threshold; (C4) an Output Semantic Gate (OSG) that detects implicit coercion using a structured $K \times I \times P$ threat calculus (Knowledge, Influence, Policy); and (C5) a formal verification framework proving that goal integrity is maintained even under adversarial model compromise. By shifting agent alignment from a behavioral property to a structurally enforced system constraint, PEA provides a robust foundation for the governance of autonomous agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。