arXiv:2505.17629cs.HCcs.AI2025-05ACL被引 5

首个评估GUI智能体跨版本、跨平台迁移能力的基准测试

TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments

  • 构建涵盖15类应用的动态界面评估基准,覆盖多版本多平台
  • 在跨版本/平台/应用场景下,接地准确率显著提升
  • 适合研究通用智能体、自动化工具与无障碍技术的开发者

图形化用户界面(GUI)智能体可通过自然语言指令自主操作数字界面,具有提升可访问性、自动化和用户体验的巨大潜力。其核心能力在于‘接地’——将语言意图映射到视觉与结构化的界面元素。然而,现有智能体难以适应真实数字环境中任务跨平台、跨应用且受版本更新影响的动态特性。为此,我们提出TransBench,首个系统评估并提升GUI智能体在三个关键维度上迁移能力的基准:跨版本迁移(适应版本更新)、跨平台迁移(在iOS、Android、Web间泛化)以及跨应用迁移(处理功能各异的应用任务)。TransBench包含15个应用类别,覆盖多版本与多平台的关键页面,支持稳健评估。实验表明,接地准确率显著提升,验证了智能体在动态真实环境中的实用价值。代码与数据将公开于GitHub。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) agents, which autonomously operate on digital interfaces through natural language instructions, hold transformative potential for accessibility, automation, and user experience. A critical aspect of their functionality is grounding - the ability to map linguistic intents to visual and structural interface elements. However, existing GUI agents often struggle to adapt to the dynamic and interconnected nature of real-world digital environments, where tasks frequently span multiple platforms and applications while also being impacted by version updates. To address this, we introduce TransBench, the first benchmark designed to systematically evaluate and enhance the transferability of GUI agents across three key dimensions: cross-version transferability (adapting to version updates), cross-platform transferability (generalizing across platforms like iOS, Android, and Web), and cross-application transferability (handling tasks spanning functionally distinct apps). TransBench includes 15 app categories with diverse functionalities, capturing essential pages across versions and platforms to enable robust evaluation. Our experiments demonstrate significant improvements in grounding accuracy, showcasing the practical utility of GUI agents in dynamic, real-world environments. Our code and data will be publicly available at GitHub.

GUI智能体跨平台迁移自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。