arXiv:2605.21404cs.LG2026-05被引 5

12篇主流LLM智能体评测论文披露不全,透明度堪忧。

What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

  • 设计五维审计框架,逐项核查评测细节
  • 智能体类论文平均披露得分仅0.38(满分1.0)
  • 成本与评测环境信息普遍缺失,适合研究者参考

我们分析了十二篇知名的大型语言模型智能体评测论文,逐维度记录每篇论文实际披露的评估细节。动机源于常见困惑:两篇论文使用相同模型和基准却结果相悖,却无法判断是评测架构、采样设置、数据子集或评估器版本差异所致。许多论文发布的代码库也难以解答。本文为一次实施报告,设计了包含五个字段的审计框架(基准身份、评测工具链、推理设置、成本报告、失败分解),编写了含边界案例的评分手册,对八篇智能体类与四篇经典静态类论文进行评分。评分聚焦评测过程的披露程度而非结果正确性,不声称披露即代表可信。八篇智能体论文平均得分为0.38(满分1.0),四篇经典论文为0.66;最大差距出现在成本披露(八篇均未提供任何推理成本)和评测工具链(均未公开内容可寻址的容器镜像)。我们发布审计框架为JSON Schema、手册为Markdown、原始评分表为CSV。评分由单人一次性完成,多评审者审计是下一步方向,我们讨论其可能带来的变化。

原文摘要 · Abstract (English)

We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In many cases the published artifact does not let you answer. This paper is an implementation report on the attempt. We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown), wrote a scoring codebook with the boundary cases we hit during pilot scoring, applied it to twelve canonical papers (eight agent, four classical static), and recorded what we saw. We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result. The mean audit score across the eight agent-benchmark papers is 0.38 (out of 1.0), and across the four classical static benchmarks 0.66; the largest gap is on cost (none of the eight agent benchmark papers disclose inference cost in any form) and on harness specification (none fully disclose a content-addressed container image of the evaluation environment). We release the schema as a JSON Schema file, the codebook as a Markdown document, and the raw scoring sheet as a CSV. The scoring was performed by a single auditor in one pass; a multi-rater audit is the natural next step, and we discuss what we think it would change.

评测透明度LLM智能体审计框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。