arXiv:2605.14133cs.AI2026-05被引 1

构建可执行的命令行交互基准,测试代理在状态冲突下的真实表现。

ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

论文配图:ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
图 1 · 摘自论文原文
  • 基于场景模板与初始状态生成可复现的任务,支持持续工作流评估。
  • 顶尖模型严格准确率仅45.3%,多数模型无法正确处理已有状态。
  • 揭示了代理在状态冲突下不同失败模式,适合评估智能体鲁棒性。

交互式智能体评测面临可扩展性与真实流程评估之间的矛盾。人工编写任务成本高且难维护,而静态提示评测无法捕捉代理在持久状态下的失效行为。现有互动基准虽显著推进了评测能力,但大多从干净状态启动,未系统检验代理对已有部分、过期或冲突产物的处理能力。本文提出 extbf{ClawForge},一个基于生成器的框架,用于在状态冲突下生成可执行的命令行工作流评测。该框架将场景模板、占位符、初始状态、参考轨迹和验证器整合为可复现的任务规范,通过标准化终态和可观测副作用进行逐步评估,而非精确匹配轨迹。我们将其实例化为ClawForge-Bench(17个场景,6类能力)。对七款前沿模型的测试显示,最优模型严格准确率仅为45.3%,错误状态替换率均低于17%,模型间最大差距(17%至90%)由是否检查现有状态决定。部分得分与步骤效率分析进一步表明,多数失败是近似完成而非早期崩溃,且模型在状态冲突下呈现质性不同的失败模式。

原文摘要 · Abstract (English)

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents operate over persistent state. Existing interactive benchmarks have advanced agent evaluation significantly, but most initialize tasks from clean state and do not systematically test how agents handle pre-existing partial, stale, or conflicting artifacts. We present \textbf{ClawForge}, a generator-backed benchmark framework for executable command-line workflows under state conflict. The framework compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications, and evaluates agents step by step over persistent workflow surfaces using normalized end state and observable side effects rather than exact trajectory matching. We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories). Results across seven frontier models show that the best model reaches only 45.3% strict accuracy, wrong-state replacement remains below 17\% for all models, and the widest model separation (17% to 90%) is driven by whether agents inspect existing state before acting. Partial-credit and step-efficiency analyses further reveal that many failures are near-miss closures rather than early breakdowns, and that models exhibit qualitatively different failure styles under state conflict.

智能体评测命令行状态冲突基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。