arXiv:2602.11749cs.AI2026-02中稿 · ICML被引 6

为大模型智能体设计了首个自动应急响应框架,提升故障应对能力。

AIR: Improving Agent Safety through Incident Response

  • 用领域语言定义应急流程,嵌入执行循环中实现自主响应。
  • 三种代理类型测试中,检测、修复、根除成功率均超90%。
  • 适合关注智能体安全的开发者与部署者,尤其在高风险场景。

大型语言模型(LLM)智能体正广泛应用于各类自主系统。然而现有安全机制主要聚焦于事前预防,对事故后的检测、控制与恢复能力有限。本文提出AIR,首个面向LLM智能体系统的应急响应框架。AIR定义了用于管理应急生命周期的领域特定语言,并集成到智能体执行循环中:(1) 基于当前环境状态和近期上下文进行语义检查以检测事故;(2) 通过工具引导智能体执行遏制与恢复操作;(3) 在消除阶段合成防护规则,防止未来重复发生类似问题。在三种代表性智能体类型上的评估显示,AIR在检测、补救和根除方面的成功率均超过90%。大量实验验证了关键设计组件的必要性,证明AIR响应及时且开销适中,并表明由LLM生成的规则可接近开发者编写的规则在跨领域中的有效性。结果表明,应急响应作为提升智能体安全性的核心机制,既可行又不可或缺。

原文摘要 · Abstract (English)

Large Language Model (LLM) agents are increasingly deployed in practice across a wide range of autonomous applications. Yet current safety mechanisms for LLM agents focus almost exclusively on preventing failures in advance, providing limited capabilities for responding to, containing, or recovering from incidents after they inevitably arise. In this work, we introduce AIR, the first incident response framework for LLM agent systems. AIR defines a domain-specific language for managing the incident response lifecycle autonomously in LLM agent systems, and integrates it into the agent's execution loop to (1) detect incidents via semantic checks grounded in the current environment state and recent context, (2) guide the agent to execute containment and recovery actions via its tools, and (3) synthesize guardrail rules during eradication to block similar incidents in future executions. We evaluate AIR on three representative agent types. Results show that AIR achieves detection, remediation, and eradication success rates all exceeding 90%. Extensive experiments further confirm the necessity of AIR's key design components, show the timeliness and moderate overhead of AIR, and demonstrate that LLM-generated rules can approach the effectiveness of developer-authored rules across domains. These results show that incident response is both feasible and essential as a first-class mechanism for improving agent safety.

智能体安全应急响应大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。