自动检测智能体执行轨迹中的隐藏错误，提升复杂任务可靠性。
Willful Disobedience: Automatically Detecting Failures in Agentic Traces
- 从提示词中提取行为规则，自动验证智能体轨迹是否合规。
- 在424条轨迹上发现结果达标但流程违规的问题，准确率超90%。
- 适合开发人员调试多轮智能体系统，尤其适用于客服场景。
AI智能体正越来越多地嵌入真实软件系统，通过多轮对话、工具调用和中间决策执行多步骤工作流。这些长周期的执行轨迹（即智能体轨迹）使验证变得困难。仅以结果为导向的评估会遗漏关键过程性错误，如错误的工作流路由、不安全的工具使用或违反提示指定规则的情况。本文提出AgentPex，一种基于AI的工具，用于系统化评估智能体轨迹。AgentPex从智能体提示词和系统指令中提取行为规则，并利用这些规范自动检测轨迹的合规性。我们在电信、零售和航空客户服务领域的τ²-bench数据集上，对424条轨迹进行了评估，涵盖多个模型。结果显示，AgentPex能有效区分不同模型的行为表现，并揭示出结果评分无法捕捉的规范违反问题。它还支持按领域和指标进行细粒度分析，帮助开发者规模化理解智能体的优势与短板。AgentPex源代码已开源：https://github.com/microsoft/agentpex。
原文摘要 · Abstract (English)
AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from $τ^2$-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。