评测智能体故障何时可被发现，分三个阶段评估方法有效性。
CatchBench: When Can an Agent Failure Be Caught?
- 从配置前、运行中到完成后的三阶段评测智能体故障检测能力。
- 72种方法在1187个配置上测试，多数结果无法排序且未给出明确排名。
- 强调标签生成过程透明性，避免因数据构建漏洞导致误判。
智能体故障何时可被发现？传统审计依赖记录而非方法。CatchBench将审计问题置于三种信息状态：运行前的声明配置（PRE）、运行过程中的追踪前缀（LIVE）和完整运行轨迹（POST）。现有基准通常只固定一种状态或改变遥测信息，尚无在统一任务-方法接口下同时评估三者的先例。每种状态对应不同问题，因此设置七项任务契约，各自配有独立标签与指标，而非单一排行榜。四类为证据型评估，三类为基于黄金标准的机制诊断。本发布涵盖72个参赛方法，包括规则扫描器、结构模型及来自九个模型家族（GPT、Claude、Gemini、Gemma、Llama、Qwen、DeepSeek、Mistral、Nova）的11位LLM裁判，在1187个声明配置和1162次实际运行中进行评估。多数结果无法排序：138组对比中有56组分离，其余未发布排名而保持未解状态。最显著的两项结果反而与自身数据相悖：一个规则忽略所有名称与权限，仅标记首个之后的每个能力声明，在六种配置来源之一达到完美F1值，该分数反映的是语料构建方式而非方法推理能力；我们因此拒绝了一个注入的基底，并对另一者不予赋予证据地位。可见，基准数值不可解读，除非其标签生成过程公开并验证是否存在捷径。我们报告全部细节，并在不调用模型的前提下重新生成所有排序。
原文摘要 · Abstract (English)
When can an agent failure be caught? An audit is usually limited by the record rather than by the method. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Most of the arena does not order: 56 of 138 registered contrasts separate, and the rest are published unresolved rather than ranked. The two sharpest results cut against our own data. One rule ignores every name and permission; it flags each capability declared after the first. On one of six configuration sources it reaches a perfect F1, so a score there measures how the corpus was built rather than how well a method reasons. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report both, and regenerate every ordering from released predictions with no model call.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。