arXiv:2607.11098cs.SEcs.AI2026-07被引 2

让大模型工具代理的故障可复现、可干预、可验证,提升部署可靠性。

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

论文配图:AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
图 1 · 摘自论文原文
  • 构建可干预的测试工作台,模拟12类工具故障并回放响应
  • 在5个代理上测试,最佳表现通过105/120种场景,最弱为77
  • 适合关注代理鲁棒性与部署安全性的开发者

工具调用型大模型代理通常假设所有工具正常运行。当工具超时、返回过期数据或描述被污染时,开发者需要一种受控方式复现问题、测试修复方案,并确认修复有效后再部署。我们提出AgentCheck,一个开源网页工作台,将MCP服务器转变为可干预的测试界面。AgentCheck先让代理使用真实工具并记录所有响应,再以故障注入器对响应进行扰动后重跑。匹配的工具调用从缓存中重放,后续调用则实时执行直至代理行为发散。该流程形成‘复现-干预-确认’闭环:开发者切换缓解策略,重新运行相同故障,观察失败是否消失。评估包含确定性通过/失败规则和基于LLM的判别标签,经人工标注验证。在五个代理中,最佳通过105/120种场景,最差仅77。失败多为静默但自信地使用错误输出,而非崩溃。在最弱代理上,重试机制使超时故障的成功率从最低30%提升至100%,而过期数据故障始终维持在3-4/10左右。AgentCheck使这些故障模式在部署前可复现、可比较、可验证。

原文摘要 · Abstract (English)

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

大模型代理故障注入部署验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。