为交互式智能体评估引入证据层,让评分结果更可信。
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation

- 在不改动任务和评测流程的前提下,增加可验证的执行证据要求。
- 通过三种证据标签区分成功、失败与未知情况,揭示评估不确定性。
- 适用于希望提升评测透明度的研究者与平台方。
交互式智能体基准测试通过结果检查将智能体运行映射为二元结果。当这些检查依赖表面信号或无法捕捉智能体的真实操作路径时,无法可靠判断运行是否成功。例如,任务询问是否更改了爱丽丝的配送地址,而结果检查仅验证点击了“保存”按钮,这并不能保证实际状态已变更,因为智能体可能修改了错误的记录。将此类运行视为成功会使报告得分产生误导。基准质量不仅取决于任务设计,还依赖于结果检测的可靠性。我们提出一种无需修改任务、智能体或评估器的成果证据报告层。该层包含三项功能:首先,在评分前明确每项案例所需的可验证存储产物;其次,对每个完成的运行应用锁定清单,并分配三种证据标签:证据通过、证据失败或未知;第三,报告基于证据的支持得分范围,量化因未知情况引发的不确定性。框架不隐匿、丢弃或隐藏不确定案例,而是将其显式呈现。我们在五个公开基准(ANDROIDWORLD、AGENTDOJO、APPWORLD、tau3 bench retail、MINIWOB)上评估该方法,结果揭示了多种可区分的失败模式。
原文摘要 · Abstract (English)
Interactive agent benchmarks map an agent run to a binary outcome through outcome checks. When these checks rely on surface level signals or fail to capture the agent's actual action path, they cannot reliably determine whether the run succeeded. For example, a benchmark task may ask whether Alice's shipping address was changed, while the outcome check only verifies that the agent clicked "Save." This does not guarantee that the intended state change occurred, since the agent may have modified the wrong record. Treating such a run as successful therefore makes the reported score misleading. Benchmark quality thus depends not only on task design, but also on the reliability of outcome detection. We address this problem by introducing an outcome evidence reporting layer for existing benchmarks, without modifying their tasks, agents, or evaluators. The layer performs three functions. First, before scoring, it specifies which stored artifacts are required to verify the claimed outcome for each case. Second, it applies a locked checklist to each completed run and assigns one of three evidence labels: Evidence Pass, Evidence Fail, or Unknown. Third, it reports evidence supported score bounds that quantify uncertainty arising from Unknown cases. Rather than silently counting, discarding, or hiding uncertain cases inside a single aggregate success rate, the framework keeps them explicitly visible. We evaluate the outcome evidence layer on five public benchmarks: ANDROIDWORLD, AGENTDOJO, APPWORLD, tau3 bench retail, and MINIWOB. The resulting reports separate several empirically distinct failure modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。