通过真实故障注入，构建了可定位LLM代理系统运行时错误的基准测试
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
- 构建包含275条执行轨迹的基准，模拟十类运行时故障
- 单条轨迹诊断准确率仅13.6%-24.8%，关键故障如绕过安全机制仍难识别
- 提供带标签的参考轨迹，支持对比评估各类诊断方法
LLM驱动的代理系统可靠性取决于完整执行过程（工具调用、模型调用、防护机制及代理间通信），而非仅最终答案。仅评估任务结果难以揭示失败原因。本文提出AGENTCHAOSBENCH，一个从执行遥测数据中检测与定位运行时故障的基准。我们运行五个异构应用，通过Agent-to-Agent协议协调代理，并通过Model Context Protocol调用工具，在工具、模型、防护机制及代理间边界注入十类操作故障（如工具不可用或响应延迟、响应损坏或过大、委托延迟、循环或错位、防护机制被绕过），同时保留无故障对照组。共生成275条经脱敏的执行轨迹：250条含故障，涵盖十类故障类型；25条无故障对照。每条故障轨迹均与同输入的无故障执行对齐，故障类型和位置标签在诊断阶段隐藏。在结构化单轨迹输入下，零样本大模型基线表现有限：最大14B参数的本地检测器故障类型识别准确率仅13.6%-19.2%，前沿模型DeepSeek-v4-pro最高24.8%；联合识别故障类型与位置最高仅22%。依赖参考的故障（尤其防护机制被绕过）仍难以解决。有参考轨迹可改善部分相对故障，但无法彻底解决防护绕过问题。该数据集与封闭预测格式支持可复现的基于大模型与非大模型诊断方法的对比。
原文摘要 · Abstract (English)
Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。