为评估代码智能体的编码能力，提出可比性更强的基准测试工具链。
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
- 设计统一的提示、运行时预算和代码提交协议，让不同智能体可公平比较。
- 在350个跨语言任务中，适配器改进使代码通过率从19.1%提升至73.4%。
- 同时关注性能与调用成本，适合需要高效评估智能体的开发者使用。
通用智能体如OpenClaw正被广泛用于自主工具操作，但其在SWE-bench上的编码能力难以衡量，因通用智能体无法满足清洁Docker工作区、补丁生成和预测合约等评分要求。本文提出Claw-SWE-Bench，一个支持多语言的类SWE-bench基准与适配协议，使异构智能体（称作claws）可在固定提示、运行预算、工作区合约、补丁提取流程及评估器下进行公平对比。完整基准包含8个语言、43个仓库的350个GitHub问题修复实例，源自SWE-bench-Multilingual与SWE-bench-Verified-Mini经未来提交清理后数据。另提供轻量版Claw-SWE-Bench Lite，由17个校准列上成本与排名感知筛选出的80个实例组成。实验显示，同一GLM 5.1骨干模型下，仅用最小直接差分适配器的OpenClaw Pass@1仅为19.1%，而完整适配器达73.4%，表明适配器设计至关重要。在一次九模型×OpenClaw与五爪×两模型的组合测试中，模型选择影响Pass@1达29.4个百分点，智能体选择影响27.4个百分点；相同准确率系统间总API成本差异显著。因此，Claw-SWE-Bench将智能体架构与成本核算作为核心评估维度，提供完整基准与低成本参考集，支持可复现的对比。数据见https://github.com/opensquilla/claw-swe-bench 及 https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench。
原文摘要 · Abstract (English)
General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. We also release Claw-SWE-Bench Lite for faster validation, which is an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns. On the full benchmark, OpenClaw with a minimal direct-diff adapter scores only $19.1\%$ Pass@1, whereas the full adapter reaches $73.4\%$ with the same GLM 5.1 backbone, showing that adapter design is essential for enabling OpenClaw-style harnesses to perform coding tasks effectively. Across an OpenClaw $\times$ nine-model sweep and a five-claw $\times$ two-model sweep, model choice changes Pass@1 by $29.4$ pp and harness choice by $27.4$ pp under fixed models; systems with similar accuracy can differ substantially in total API cost. Claw-SWE-Bench therefore treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation, providing both a full benchmark and a low-cost reference set for reproducible comparison. The data is available at https://github.com/opensquilla/claw-swe-bench and https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。