arXiv:2605.12131cs.AI2026-05

为智能体研究建立可复现的滚动记录标准,解决评估结果不可靠问题。

Rollout Cards: A Reproducibility Standard for Agent Research

论文配图:Rollout Cards: A Reproducibility Standard for Agent Research
图 1 · 摘自论文原文
  • 用滚动卡片封装完整执行记录与评估规则,取代仅报告分数
  • 同一任务在不同规则下得分差达20.9个百分点,甚至反转模型排名
  • 适用于工具使用、代码生成、多智能体等前沿智能体研究领域

智能体研究中的可复现性问题日益突出:论文仅报告分数,而隐藏了关键的滚动记录。我们对50个主流训练与评估仓库进行结构化审计,发现均未披露失败、出错或跳过的运行次数。同时记录到37种报告规则变化会显著影响任务成功率、成本/令牌计算或时间测量,即使证据不变。我们提出以滚动记录为可复现性单位,引入滚动卡片——包含完整执行记录、评估视角、规则说明和数据剔除清单的发布包。在工具安全、多智能体系统、定理证明和搜索任务中验证,部分公开释放的卡片使原报告无法完成的分析成为可能。跨任务重新评分显示,仅改变报告规则即可导致分数变动20.9绝对百分点,部分情况下反转领先模型排名。我们开源了集成于Ergon(一个开源强化学习环境)的参考实现,并公开发布覆盖工具使用、软件工程、网络交互、多智能体协作、安全性和搜索任务的滚动卡片输出,支持未来研究。

原文摘要 · Abstract (English)

Reproducibility problems that have long affected machine learning and reinforcement learning are now surfacing in agent research: papers compare systems by reported scores while leaving the rollout records behind those scores difficult to inspect. For agentic tasks, this matters because the same behaviour can receive different reported scores when evaluations select different parts of a rollout or apply different reporting rules. In a structured audit of 50 popular training and evaluation repositories, we find that none report how many runs failed, errored, or were skipped alongside headline scores. We also document 37 cases where reporting rules can change task-success rates, cost/token accounting, or timing measurements for fixed evidence, sometimes dramatically. We treat rollout records, not reported scores, as the unit of reproducibility for agent research. We introduce rollout cards: publication bundles that preserve the rollout record and declare the views, reporting rules, and drops manifests behind reported scores. We validate rollout cards in two settings. First, four partial public releases in tool safety, multi-agent systems, theorem proving, and search let us compute analyses their original reports did not include. Second, re-grading preserved benchmark outputs across short-answer, code-generation, and tool-use tasks shows that changing only the reporting rule can change reported scores by 20.9 absolute percentage points and, in some cases, invert rankings of frontier models. We release a reference implementation integrated into Ergon, an open-source reinforcement learning gym, and publicly publish Ergon-produced rollout-card exports for benchmarks spanning tool use, software engineering, web interaction, multi-agent coordination, safety, and search to support future research.

智能体研究可复现性评估标准滚动记录

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。