首个面向macOS的多语言交互评测基准，填补了GUI代理在苹果系统上的空白。
macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
- 构建202个跨30款应用的多语言任务，含28款macOS独占应用。
- 开源模型成功率不足5%，大厂闭源模型超30%，差距显著。
- 发现阿拉伯语任务平均降级28.8%，且欺骗攻击普遍存在。
图形用户界面（GUI）代理在自动化计算机操作和辅助无障碍使用方面展现出巨大潜力，但现有交互评测基准多为英文，覆盖网页、Windows、Linux和Android环境，未涵盖macOS。macOS拥有独特的界面模式和专属应用。为此，我们提出macOSWorld，首个针对macOS的综合性GUI代理评测基准。该基准包含202个多语言交互任务，覆盖30个应用程序（其中28个为macOS独占），任务指令与操作系统界面支持5种语言（英语、中文、阿拉伯语、日语、俄语）。由于GUI代理易受欺骗攻击，macOSWorld还包含专门的安全评测子集。对六款GUI代理的评估显示：商业级代理成功率超过30%，而开源轻量级研究模型低于5%，凸显亟需针对macOS领域进行适应。多语言评测也暴露出普遍弱点，尤其在阿拉伯语中，平均性能相比英语下降28.8%。安全评测结果进一步表明，欺骗攻击具有广泛性，亟需关注。项目页面：https://macos-world.github.io。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android environments, but not macOS. macOS is a major OS with distinctive GUI patterns and exclusive applications. To bridge the gaps, we present macOSWorld, the first comprehensive benchmark for evaluating GUI agents on macOS. macOSWorld features 202 multilingual interactive tasks across 30 applications (28 macOS-exclusive), with task instructions and OS interfaces offered in 5 languages (English, Chinese, Arabic, Japanese, and Russian). As GUI agents are shown to be vulnerable to deception attacks, macOSWorld also includes a dedicated safety benchmarking subset. Our evaluation on six GUI agents reveals a dramatic gap: proprietary computer-use agents lead at above 30% success rate, while open-source lightweight research models lag at below 5\%, highlighting the need for macOS domain adaptation. Multilingual benchmarks also expose common weaknesses, especially in Arabic, with a 28.8% average degradation compared to English. Results from safety benchmarking also highlight that deception attacks are more general and demand immediate attention. Project page: https://macos-world.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。