工具演化智能体的验证与合规之间存在差距，仅靠任务完成率无法发现。
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
- 通过保留工具源码并回放测试，检测合成工具在未见输入下的正确性
- 96.8%的合成工具在独立测试中出现错误，尽管运行时无报错
- 提出审计框架，支持部署后监控与可复现性验证
能够自主生成工具的智能体在输出答案的同时，还生成了未来任务可复用、组合和依赖的软件库。任务完成率（TC）仅验证答案正确性，不验证库的可靠性。在Claude Haiku 4.5的试点中，我们改进了测试框架以保留每个工具的源码，并对222个合成工具进行独立合规性回放测试。三个协议下，96.8%的工具在持留输入上返回错误答案：两个协议无声失效率达100%，一个为91.7%。人工编写参考实现则在全部16个能力套件中得分均为1.00。合成工具运行顺畅，会话内验证器未触发警告，但在未见输入下表现异常。这一验证-合规性差距是传统通过率指标无法察觉的。EvolveTool-Bench使该差距可测量：会话分种子、间隙、变体、组合、回归和对抗角色；输出包括每工具清单、经验证的TC、正确/错误重用记录及审计轨迹。在五协议试点（3种子、8会话、每轮51次验证决策）中，仅任务完成率无法区分协议，而审计层可区分。预设对比显示，适应性决策分裂合成协议表现低于一次性合成（-7.1个百分点，95%置信区间[-13.9,-0.1]，校正后不显著）——提示潜在假设，非确定结论。我们公开测试框架、工具源码保存、合规性回放及可复现性清单。此类审计结构——验证覆盖率、重用分解、持留合规性——是部署中的工具演化智能体应提供的最低输出标准。
原文摘要 · Abstract (English)
Agents that synthesize their own tools ship a second artifact alongside each answer: a software library that future tasks reuse, compose, and depend on. Task completion (TC) certifies the answer; it does not certify the library. On a Claude Haiku 4.5 pilot, we patch the harness to preserve per-tool source and replay every synthesised tool against a held-out conformance suite. Across 222 preserved tools and three protocols, 96.8% record per-tool correctness C=0.00: two protocols silent-rot at 100%, one at 91.7%. Hand-written reference implementations score C=1.00 on all 16 capability suites. The synthesised tools execute cleanly and the in-session verifier raises no flag; on held-out inputs they simply return wrong answers. This is the verification-vs.-conformance gap a pass-rate benchmark cannot see. EvolveTool-Bench makes that gap measurable. Sessions are structured into seed, gap, variant, composition, regression, and adversarial roles; runs emit per-tool manifests, verified-subset TC, correct-vs.-incorrect reuse, and audit traces for post-deployment monitoring. In the Haiku pilot over five protocols (3 seeds, 8 sessions, 51 verified decisions per pass), TC alone does not separate the protocols; the audit layer does. One pre-specified contrast shows the adapted decision-split synthesis protocol underperforming one-shot synthesis (-7.1 pp, 95% CI [-13.9,-0.1]; not significant after BH correction) -- a hypothesis-generating signal, not a confirmed effect. We release the harness, per-tool source preservation, conformance replay, and reproducibility manifests. An audit schema of this shape -- verifier coverage, reuse decomposition, held-out conformance -- is the minimum a deployed tool-evolving agent should emit between releases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。