构建首个面向 macOS 桌面的综合性智能体评测基准，覆盖多应用交互任务。
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
- 设计包含676个任务的基准，近六成需同时操作GUI与CLI。
- 最佳模型在1次尝试下通过率73.7%，主要依赖技能库而非框架设计。
- 引入细粒度评分机制，揭示模型间子目标完成率差异。
桌面自动化智能体（CUAs）发展迅速，越来越多用户在 Mac Mini 上部署 OpenClaw 等工具实现持续自动化。然而，现有基准大多未考虑框架增强，且仅使用二值评估，无法反映现代 CUAs 所依赖的框架能力，也难以衡量长周期、多应用任务中的部分进展。为此，我们提出 MacAgentBench，一个涵盖25个应用、共676个任务的综合性 macOS 评测基准，近60%的任务涉及图形界面与命令行交互。该基准采用确定性规则评估，并引入细粒度多检查点评分与能力标注，支持多应用任务分析。在三个框架和16个模型上的实验表明，最佳配置（Claude Opus 4.6 + OpenClaw）达到73.7% Pass@1，此优势主要来自技能库而非框架设计。细粒度指标进一步显示，具有相似总通过率的模型在子目标完成上存在显著差异。代码与数据已开源于 https://github.com/JetAstra/MacAgentBench。
原文摘要 · Abstract (English)
Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。