为自主智能体设计实时验证框架，动态评估其行为出错概率。
AgentGuard: Runtime Verification of AI Agents
- 通过观察输入输出抽象为状态转移事件，构建动态更新的马尔可夫决策过程。
- 利用概率模型检测实现实时定量验证，持续提供失败概率保障。
- 适合关注AI系统安全性的研发与运维人员，尤其在高风险场景中。
自主智能体系统的快速发展带来了显著风险，因其内在不可预测性和涌现行为，传统验证方法已不适用，亟需转向基于概率保证的新范式——问题不再是谁会失败，而是失败的概率是多少。本文提出AgentGuard，一种面向智能体系统的运行时验证框架，通过动态概率保证新范式提供持续、量化的可信保障。AgentGuard作为检测层，观测智能体原始输入/输出，并将其抽象为状态模型中的形式化事件；随后利用在线学习动态构建并更新一个马尔可夫决策过程（MDP），以形式化建模智能体的涌现行为；最后通过概率模型检测，在线实时验证定量性质。
原文摘要 · Abstract (English)
The rapid evolution to autonomous, agentic AI systems introduces significant risks due to their inherent unpredictability and emergent behaviors; this also renders traditional verification methods inadequate and necessitates a shift towards probabilistic guarantees where the question is no longer if a system will fail, but the probability of its failure within given constraints. This paper presents AgentGuard, a framework for runtime verification of Agentic AI systems that provides continuous, quantitative assurance through a new paradigm called Dynamic Probabilistic Assurance. AgentGuard operates as an inspection layer that observes an agent's raw I/O and abstracts it into formal events corresponding to transitions in a state model. It then uses online learning to dynamically build and update a Markov Decision Process (MDP) that formally models the agent's emergent behavior. Using probabilistic model checking, the framework then verifies quantitative properties in real-time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。