构建新数据集与框架，提升跨应用的GUI智能体泛化能力
GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
- 通过探索-推理框架构建多任务数据集，增强跨应用迁移能力
- 在陌生环境中比现有方法提升10%性能，验证泛化效果
- 适合研究GUI智能体、人机交互与通用机器人任务的开发者
GUI智能体在提升人机交互体验与效率方面潜力巨大，但现有方法在跨应用和跨任务泛化上面临挑战，主要源于现有数据集的两大局限：一是忽略开发者引入的应用间结构差异，限制知识迁移；二是仅聚焦导航任务，难以反映复杂软件架构与用户交互。为此，我们提出GUI-Xplore，一个精心设计的数据集，通过预录探索视频提供上下文信息，并包含五个分层的下游任务，全面评估GUI智能体能力。为充分挖掘该数据集价值，我们提出Xplore-Agent框架，结合动作感知的GUI建模与图引导的环境推理。实验表明，Xplore-Agent在陌生环境中相较现有方法提升10%，但仍存在显著优化空间，以实现真正通用的GUI智能体。
原文摘要 · Abstract (English)
GUI agents hold significant potential to enhance the experience and efficiency of human-device interaction. However, current methods face challenges in generalizing across applications (apps) and tasks, primarily due to two fundamental limitations in existing datasets. First, these datasets overlook developer-induced structural variations among apps, limiting the transferability of knowledge across diverse software environments. Second, many of them focus solely on navigation tasks, which restricts their capacity to represent comprehensive software architectures and complex user interactions. To address these challenges, we introduce GUI-Xplore, a dataset meticulously designed to enhance cross-application and cross-task generalization via an exploration-and-reasoning framework. GUI-Xplore integrates pre-recorded exploration videos providing contextual insights, alongside five hierarchically structured downstream tasks designed to comprehensively evaluate GUI agent capabilities. To fully exploit GUI-Xplore's unique features, we propose Xplore-Agent, a GUI agent framework that combines Action-aware GUI Modeling with Graph-Guided Environment Reasoning. Further experiments indicate that Xplore-Agent achieves a 10% improvement over existing methods in unfamiliar environments, yet there remains significant potential for further enhancement towards truly generalizable GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。