新基准通过运行时追踪评估智能体，揭示传统榜单忽略的失败模式。
ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts
- 基于运行时轨迹设计双轨评测：实时工作区任务与封闭式快照任务
- 顶层安全门控得分0.7671，真实任务表现远低于理想值
- 适合关注智能体运行可靠性、安全性及过程质量的研究者
传统智能体评测仅关注最终答案，忽视状态化运行时中的执行过程。本文提出ClawProBench，一个面向OpenClaw运行时的追踪感知基准，包含102个场景的完整配置和68个场景的冻结留出集。前者支持实时工作区与原生路由任务，后者采用封闭世界JSON输出契约以实现鲁棒排序。评分基于执行轨迹，结合正确性、过程质量与效率，设有安全门控机制并保留故障证据用于审计。在完整配置上评估68种配置，在留出集上评估37种。最高安全门控平均得分为0.7671；原生运行时任务表现（0.5238）显著低于工作区任务（0.6415）。在留出集上，pass@k-any（0.6638）优于严格三轮通过率（0.2890），但完整配置与留出集排名相关性极低（Spearman 0.1300）。仅依据正确性的排名与过程感知、安全门控或严格通过视图差异显著。最终答案排行榜可能掩盖原生界面缺陷、偶然成功及局部失败模式。
原文摘要 · Abstract (English)
Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。