构建可复现的移动端GUI代理评估基准，解决真实与可控间的矛盾
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 用模拟应用+人工验证生成可控环境，保留真实交互逻辑
- 19个模型测试仅50.27%任务完成，28.55%任务无一解决
- 适合研究移动智能体、自动化测试及可复现评估的团队
移动端GUI代理能根据自然语言指令执行任务，但其评估难以兼顾真实性和可复现性。现有基准常在二者间妥协：简化应用缺乏真实复杂度，而真实商业应用则因推荐、广告、账户和内容变动引入不可控变量。我们提出AppSim-Bench，通过可控模拟应用解决这一矛盾，保留任务相关交互逻辑并支持确定性评估。该基准基于编码代理辅助与人工验证的流程构建，涵盖17个高频中英文应用中的557项任务。其可控后端数据与基于结果的验证机制消除了主要环境随机性，实现跨模型可复现比较。对19个GUI代理（包括通用型与专用型）的评估表明，自主移动端执行仍未解决：最优模型仅完成50.27%任务，28.55%任务未被任何代理解决。进一步分析显示，失败集中在长流程、数值推理任务，以及高动作开销和预算耗尽的低效轨迹上。项目开源地址：https://github.com/Acrab-Agentic-Labs/AppSim。
原文摘要 · Abstract (English)
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。