构建首个面向移动端的混合操作评估基准，评测智能体自动生成快捷方式的能力。
MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents
- 提出自动生成可复用快捷方式的新方法，突破传统预设路径限制。
- 在11个真实应用上测试，混合智能体成功率最高达68.3%，效率提升39%。
- 适合研究移动自动化、智能体决策与低代码生成的开发者和学者。
快捷方式如API和深度链接已成为灵活图形界面操作的有效补充，推动了基于多模态大模型的移动端自动化发展。然而，对图形界面-快捷方式混合智能体的系统性评估仍不充分。为此，我们提出MAS-Bench，首个聚焦移动端的混合操作评估基准。该基准不仅使用预定义快捷方式，更评估智能体自主发现并创建可复用、低成本工作流的能力。包含139个复杂任务、11个真实应用、88个预定义快捷方式（含API、深度链接、RPA脚本）以及9项评估指标。实验表明，混合智能体成功率最高达68.3%，执行效率比纯图形界面方案高39%。评估框架有效揭示了预定义快捷方式与智能体生成快捷方式之间的质量差距，验证其评估生成能力的有效性。MAS-Bench填补了移动端混合操作智能体缺乏系统性评估基准的空白，为未来高效、鲁棒智能体的发展提供基础平台。
原文摘要 · Abstract (English)
Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, fostering a promising hybrid paradigm for MLLM-based mobile automation. However, systematic evaluation of GUI-shortcut hybrid agents remains largely underexplored. To bridge this gap, we introduce MAS-Bench, a benchmark that pioneers the evaluation of GUI-shortcut hybrid agents with a specific focus on the mobile domain. Beyond merely using predefined shortcuts, MAS-Bench assesses an agent's capability to autonomously generate shortcuts by discovering and creating reusable, low-cost workflows. It features 139 complex tasks across 11 real-world applications, a knowledge base of 88 predefined shortcuts (APIs, deep-links, RPA scripts), and 9 evaluation metrics. Experiments demonstrate that hybrid agents achieve up to 68.3% success rate and 39% greater execution efficiency than GUI-only counterparts. Furthermore, our evaluation framework effectively reveals the quality gap between predefined and agent-generated shortcuts, validating its capability to assess shortcut generation methods. MAS-Bench addresses the lack of systematic benchmarks for GUI-shortcut hybrid mobile agents, providing a foundational platform for future advancements in creating more efficient and robust intelligent agents. Project page: https://pengxiang-zhao.github.io/MAS-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。