自动诊断AI代理失败原因，精准定位问题步骤。
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- 构建170条轨迹的失败诊断基准，覆盖11类任务场景。
- 自动化框架将定位准确率提升75%，并给出失败分类。
- 适合调试复杂AI代理系统的研发人员使用。
AI代理的失败往往难以定位，因其执行过程具有概率性、长时程、多智能体交互及工具输出噪声等特点。为此，我们人工标注了170条失败轨迹，涵盖11种不同任务场景，包括结构化API工作流、事件管理以及开放式网页/文件任务。每条轨迹均标注关键失败步骤和基于扎根理论的跨领域失败类别。为降低人工诊断成本，提出AgentRx——一个自动化诊断框架，通过逐步合成与评估约束条件，生成可审计的违规证据日志；再由大模型裁判根据日志定位关键失败步骤与类别。实验显示，该方法在步骤定位上相比先前工作平均提升75%，同时实现失败类别归因。
原文摘要 · Abstract (English)
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an $\textit{automated diagnostic framework}$ that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。