arXiv:2505.16827cs.AI2025-05ACL被引 37

无需训练即可自动探索和挖掘界面操作逻辑的智能自动化工具

GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent

  • 通过分析界面结构自动生成探索目标,系统收集多样化操作轨迹
  • 无监督提取状态转移中的操作逻辑,在SPA-Bench上达到53.7%成功率
  • 无需微调即可适配新应用,适合跨平台自动化测试场景

GUI自动化在动态环境中面临严峻挑战。大模型在理解界面组件和知识更新方面存在缺陷。传统微调成本高且难以适应新应用。我们提出GUI-explorer,一种无需训练的GUI智能体,包含两项核心机制:(1) 功能感知的自主探索轨迹生成。基于界面结构信息(如截图和活动层级)自动构建探索目标,实现对应用功能的全面覆盖,系统化收集多样化操作轨迹。(2) 转移感知知识的无监督挖掘。通过无监督分析结构化交互三元组(观察、动作、结果)的状态转移,提取精准的屏幕-操作逻辑,无需人工参与知识提取。在SPA-Bench上任务成功率达53.7%,AndroidWorld上达47.4%,显著优于现有先进方法。该模型无需参数更新即可适配新应用。项目已开源,地址为https://github.com/JiuTian-VL/GUI-explorer。

原文摘要 · Abstract (English)

GUI automation faces critical challenges in dynamic environments. MLLMs suffer from two key issues: misinterpreting UI components and outdated knowledge. Traditional fine-tuning methods are costly for app-specific knowledge updates. We propose GUI-explorer, a training-free GUI agent that incorporates two fundamental mechanisms: (1) Autonomous Exploration of Function-aware Trajectory. To comprehensively cover all application functionalities, we design a Function-aware Task Goal Generator that automatically constructs exploration goals by analyzing GUI structural information (e.g., screenshots and activity hierarchies). This enables systematic exploration to collect diverse trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise screen-operation logic, we develop a Transition-aware Knowledge Extractor that extracts effective screen-operation logic through unsupervised analysis the state transition of structured interaction triples (observation, action, outcome). This eliminates the need for human involvement in knowledge extraction. With a task success rate of 53.7% on SPA-Bench and 47.4% on AndroidWorld, GUI-explorer shows significant improvements over SOTA agents. It requires no parameter updates for new apps. GUI-explorer is open-sourced and publicly available at https://github.com/JiuTian-VL/GUI-explorer.

GUI自动化无监督学习智能代理状态转移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。