构建真实环境下的长时序智能体评估基准，检验模型在实际运行中的表现。
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
- 在真实CLI环境中运行60个真人编写、多模态任务，模拟长期操作场景。
- 顶尖模型最高仅达62.2%成功率，显示当前智能体能力仍有显著差距。
- 适合关注智能体真实部署性能与评估方法的研究者使用。
大型语言和视觉-语言模型正越来越多地通过命令行接口（CLI）驱动智能体执行用户任务。然而，大多数智能体评估基准仍依赖合成沙盒、短时任务、模拟服务接口和最终答案检查，无法验证智能体在真实部署环境中的长期任务完成能力。本文提出WildClawBench，一个包含60个真人撰写的双语多模态任务的原生运行时基准，覆盖六大主题类别。每个任务平均耗时约8分钟，需超过20次工具调用，运行于可复现的Docker容器中，使用真实的CLI代理框架（OpenClaw、Claude Code、Codex或Hermes Agent），并接入真实工具而非模拟服务。评分采用混合机制：结合确定性规则检查、环境状态审计及大模型/视觉-语言模型判官进行语义验证。在19个前沿模型中，最佳表现者Claude Opus 4.7在OpenClaw下达到62.2%整体准确率，其余模型均低于60%，而更换代理框架本身可使同一模型得分变动高达18个百分点。结果表明，当前前沿模型在长时序、原生运行时环境下仍难以胜任复杂任务。我们已开源所有任务、代码与容器化工具，支持可复现评估。
原文摘要 · Abstract (English)
Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。