测试智能体在复杂依赖接口流程中的生存能力,发现失败原因远超简单执行失败。
APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

- 构建可审计的接口流程基准,分解为七项工程能力评估
- 20步依赖链成功率仅61%,多数失败在最终交付阶段
- 强调可靠性比单次表现更重要,适合高要求生产系统研发者
工具使用智能体通常仅以端到端流程是否完成这一单一指标评估。该指标无法区分生产中真正重要的失败,如凭证过期、负载格式错误,或正确执行后交付错误。我们提出APIFlow-Bench,一个完全可审计的长周期、依赖式REST-API工作流基准,将性能分解为七项工程能力,并要求智能体的回答必须由实际调用路径支持。我们正向生成合成API世界,逐子任务验证;每个子任务需通过零大模型自检三元组验证评分器,且由可信第三方确认可解性,同时对抗性审计发现了六个评分器漏洞并修复。评分确定且溯源敏感:状态检查追踪模拟生成的信标通过API数据流至响应,答案卡片按字段逐项验证。我们发布所有答案密钥及44,362条未脱敏执行日志。在19个前沿与开源模型上使用统一框架测试发现:(1) 依赖链越长成功率越低,从单个子任务的93%降至20步干净链的74%,加入8%被模型共识筛选为全模型通过的试验后降至61%;(2) 可靠性比最佳能力更能区分模型,五次采样中至少一次通过提升7个百分点,而五次全部通过的可靠性差距达44个百分点;(3) 独立错误累加模型不适用数据:20步链成功率比各子任务率乘积高出33个百分点,干净子集中77%的失败案例已抵达正确最终状态,仅在交付阶段出错。
原文摘要 · Abstract (English)
Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payloads, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent REST-API workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. We generate synthetic API worlds forward, subtask by subtask; each subtask is admitted only after a zero-LLM self-test triad verifies its grader and an oracle establishes solvability, and an adversarial audit identified and fixed six grader exploits. Grading is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response the answer must originate from, and a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts. Across 19 frontier and open-weight models under one neutral scaffold, we find: (1) longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model; (2) reliability separates models more than best-case capability, with best-of-five spanning seven points but all-five-of-five reliability spanning 44 points; (3) the independent-error account of compounding failure does not fit the data: pass rates on 20-subtask chains are 33 percentage points above the product of subtask-level rates, and on the clean slice 77% of failing runs reached the correct final state and failed only at delivery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。