为多智能体AI系统建立可形式化验证的安全与功能分析框架。
Formalizing the Safety, Security, and Functional Properties of Agentic AI Systems
- 构建主机代理与任务生命周期双模型,统一多智能体交互语义。
- 定义30项时序逻辑性质,支持死锁、安全漏洞的自动检测。
- 适用于高风险场景的AI系统设计,如医疗、金融自动化决策。
多智能体人工智能系统通过多个自主代理和大语言模型协同完成复杂多步骤任务,其安全性、可靠性与功能性在高风险应用中至关重要。当前智能体间通信协议(如MCP工具访问协议、A2A协调协议)分散孤立,导致语义鸿沟,引发架构错配与可被利用的协作缺陷。为此,本文提出一个包含两个核心模型的建模框架:(1) 主机代理模型,描述与用户交互、任务分解与执行调度的顶层实体;(2) 任务生命周期模型,刻画子任务从创建到完成的状态与转移,实现细粒度的任务管理与错误处理。二者结合形成统一语义框架,支撑对多智能体系统的行为推理。基于此框架,定义了16项主机代理属性与14项任务生命周期属性,涵盖活性、安全性、完备性与公平性,以时序逻辑表达,支持系统行为的形式化验证、协作边缘情况识别及死锁与安全漏洞预防。本研究首次构建了领域无关、严谨可靠的多智能体AI系统分析、设计与部署框架。
原文摘要 · Abstract (English)
Agentic AI systems, which leverage multiple autonomous agents and large language models (LLMs), are increasingly used to address complex, multi-step tasks. The safety, security, and functionality of these systems are critical, especially in high-stakes applications. However, the current ecosystem of inter-agent communication is fragmented, with protocols such as the Model Context Protocol (MCP) for tool access and the Agent-to-Agent (A2A) protocol for coordination being analyzed in isolation. This fragmentation creates a semantic gap that prevents the rigorous analysis of system properties and introduces risks such as architectural misalignment and exploitable coordination issues. To address these challenges, we introduce a modeling framework for agentic AI systems composed of two central models: (1) the host agent model formalizes the top-level entity that interacts with the user, decomposes tasks, and orchestrates their execution by leveraging external agents and tools; (2) the task lifecycle model details the states and transitions of individual sub-tasks from creation to completion, providing a fine-grained view of task management and error handling. Together, these models provide a unified semantic framework for reasoning about the behavior of multi-AI agent systems. Grounded in this framework, we define 16 properties for the host agent and 14 for the task lifecycle, categorized into liveness, safety, completeness, and fairness. Expressed in temporal logic, these properties enable formal verification of system behavior, detection of coordination edge cases, and prevention of deadlocks and security vulnerabilities. Through this effort, we introduce the first rigorously grounded, domain-agnostic framework for the analysis, design, and deployment of correct, reliable, and robust agentic AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。