构建可执行基准测试套件,明确区分任务、动作生成与证据认定。
An Executable Benchmarking Suite for Tool-Using Agents
- 通过统一的证据准入协议,显式分离任务、驱动器与验证证据。
- 在相同条件下,不同评估策略选出不同控制器,证明证据规则影响决策。
- 适合关注评估透明性与可复现性的工具型智能体研究者。
闭环工具使用智能体在可执行的网页、代码和微任务环境中评估日益增多,但基准报告常混淆工作负载、动作生成驱动与系统声称所接受的证据。我们提出一个可执行的基准测试套件,通过共享的证据准入契约显式定义这些对象。该套件连接WebArena Verified(SWE-Gym的一个子集,兼容SWE-bench验证)、MiniWoB++,通过通用的工作负载适配器、任务清单、事件模式、重播/冻结策略、声明的驱动器和报告流水线实现互操作。在标准发布版本中,门控机制将面向论文的证据与预检、夹具、烟雾测试及诊断行分隔,同时保留未被采纳的产物用于审计与入门。被采纳的证据记录延迟、无效动作行为、补丁生成成本、验证器元数据、重播绑定和溯源信息,均受单一可审计契约约束。该门控具有决策相关性:在另一项WebArena Verified控制器研究中,干净基线与中度负载压力评估在相同工作负载和准入契约下选择了不同的固定控制器变体。本发布仅限于基准套件与被采纳的证据,不涉及新智能体策略、模型排行榜、后端对比或自主SWE-bench求解器。
原文摘要 · Abstract (English)
Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims. We present an executable benchmarking suite that makes these objects explicit under a shared evidence-admission contract. The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through common workload adapters, task manifests, event schemas, replay/freeze policy, declared drivers, and reporting pipelines. In the canonical release, the gate separates paper-facing evidence from preflight, fixture, smoke, and diagnostic rows while preserving non-admitted artifacts for audit and onboarding. The admitted evidence records latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract. The gate is decision-relevant rather than merely clerical: in a separate WebArena Verified controller study, clean-baseline and medium live-stressed evaluation select different fixed controller variants under the same workload and admission contract. The release is scoped as a benchmarking suite and admitted evidence, not a new agent policy, model leaderboard, backend comparison, or autonomous SWE-bench solver.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。