arXiv:2608.02444cs.AI2026-08中稿 · the 2026 ACM Inter…

提出决策层,让部分评估也能可靠判断模型优劣。

ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

  • 引入决策层,基于已运行任务动态判断模型优劣
  • 仅需15%~25%任务结果即可达成与完整评估一致结论
  • 适合需要快速评估且关注决策可信度的研究者

LLM-agent评估常在完整基准测试完成前就得出任务结果。仅报告部分得分不可靠,因早期任务可能遗漏关键信息,优先运行简单任务会扭曲样本分布,且仅判定易比对项的规则看似准确,实则留下大量未决比较。我们提出ParEvalLayer,一个读取双系统配对结果与预设比较策略的决策层。对每次部分运行,它判断当前结果是否已足够支持‘某系统更优’或‘不更优’,或需更多证据、或应弃权。通过重播公开基准数据,模拟提前终止评估,在每一步仅使用已有结果应用策略;若达成两类判断之一,则检验其是否与完整评估结果一致。主比较规则下,三个公开基准在仅完成15%至25%任务后即达一致结论;其余基准需更多任务。这一差异说明:仅报告部分分数不足,必须同时说明决策规则及剩余未决比较数。

原文摘要 · Abstract (English)

LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.

模型评估决策层部分评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。