AI代理安全应像运行时契约一样实时管控,防止危险行为并验证正确操作。
Agent Safety Should Be a Runtime Contract
- 用沙盒、权限门和轨迹监控提前阻止危险动作
- 通过测试运行、日志捕获等证据验证任务执行真实性
- 适合关注自主智能体安全的开发者与研究者
当前主流将AI安全视为训练阶段通过强化学习人类反馈(RLHF)、直接偏好优化(DPO)或宪法AI植入的属性。我们主张,对于执行代码、修改文件、发送消息、操作数据库的自主智能体而言,这种做法在结构上不足。安全应作为运行时契约,由执行框架强制实施,包含预防和证据两个层面:预防面通过沙盒、权限门、输出过滤器和轨迹监控阻止危险行为;证据面要求可验证的执行证据,如测试运行、日志捕获、文件差异和引用溯源,作为任务提交的前置条件。本文基于四项公开证据支持该观点:对52起已记录的智能体安全事件的调查、31个无争议核心案例加1个争议示例的错误完成审计、对12个公开代理系统与框架的轨迹模式审计,以及对NeurIPS、ICML、ICLR 2023-2025年共28,560篇录用论文的标题级审计,显示训练期与部署期发表量存在8-12倍不平衡。计算机安全与实验科学曾面临类似挑战,均采用兼具预防与证据的运行时契约;如今智能体AI也处于相同压力之下。我们形式化了代理轨迹模式与证据链,提出基于标准监控组合的组合式门控命题,并勾勒研究路线。真正的安全单位是‘带可验证证据的轨迹’,而非模型本身。
原文摘要 · Abstract (English)
The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。