用时序逻辑监控智能体行为,发现模型更新后的问题。
An Approach to Checking Correctness for Agentic Systems
- 用动作序列而非文本匹配来检测智能体异常
- 小模型导致工具调用顺序错误,被及时识别
- 适合开发验证和上线后持续监测的团队
本文提出一种时序表达语言,用于监控基于大模型的智能体系统行为,解决因随机生成导致输出不一致带来的错误检测难题。该方法借鉴硬件验证中的时序逻辑技术,关注智能体工具调用和状态转移的执行轨迹,而非具体文本输出。通过定义跨多种场景的行为正确性断言,可验证提示工程与安全约束效果,并支持模型升级或逻辑修改后的回归测试。在三智能体协作求解多步推理任务的实验中,大模型版本均满足所有时序断言;而当其中两个智能体替换为小模型时,出现工具调用顺序错误和协作交接失败,导致行为断言被违反。时序表达式成功识别出这些异常,验证了该方法在生产环境中检测行为退化的有效性。为关键应用中智能体系统的可靠性监控提供了系统性基础。
原文摘要 · Abstract (English)
This paper presents a temporal expression language for monitoring AI agent behavior, enabling systematic error-detection of LLM-based agentic systems that exhibit variable outputs due to stochastic generation processes. Drawing from temporal logic techniques used in hardware verification, this approach monitors execution traces of agent tool calls and state transitions to detect deviations from expected behavioral patterns. Current error-detection approaches rely primarily on text matching of inputs and outputs, which proves fragile due to the natural language variability inherent in LLM responses. The proposed method instead focuses on the sequence of agent actions -- such as tool invocations and inter-agent communications -- allowing verification of system behavior independent of specific textual outputs. The temporal expression language provides assertions that capture correct behavioral patterns across multiple execution scenarios. These assertions serve dual purposes: validating prompt engineering and guardrail effectiveness during development, and providing regression testing when agents are updated with new LLMs or modified logic. The approach is demonstrated using a three-agent system, where agents coordinate to solve multi-step reasoning tasks. When powered by large, capable models, all temporal assertions were satisfied across many test runs. However, when smaller models were substituted in two of the three agents, executions violated behavioral assertions, primarily due to improper tool sequencing and failed coordination handoffs. The temporal expressions successfully flagged these anomalies, demonstrating the method's effectiveness for detecting behavioral regressions in production agentic systems. This approach provides a foundation for systematic monitoring of AI agent reliability as these systems become increasingly deployed in critical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。