用分层多模态技能提升GUI代理的长任务执行能力
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills
- 构建分层多模态技能体系，从轨迹抽象出元技能
- 引入增强型蒙特卡洛树搜索，缩小在线探索动作空间
- 在多个真实场景中表现优于现有方法，适合跨平台自动化
近期将多模态大语言模型（MLLM）作为图形用户界面（GUI）代理的研究取得了良好进展。然而，这些代理在在线环境中的长周期任务上仍面临挑战，主要源于知识不足以及离线与在线域之间的固有差异。受人类在开放环境中知识泛化方式的启发，本文提出层级多模态技能（HMS）模块，通过逐步将轨迹抽象为执行技能、核心技能和元技能，构建用于长周期任务规划的分层知识结构。为弥合域间差距，提出技能增强型蒙特卡洛树搜索（SA-MCTS）算法，在线探索时有效利用离线习得的技能以压缩动作搜索空间。基于HMS，我们构建了Mirage-1——一个支持多模态、跨平台、即插即用的GUI代理。为验证其在真实长周期场景下的性能，我们构建了新基准AndroidLH。实验结果表明，Mirage-1在AndroidWorld、MobileMiniWob++、Mind2Web-Live和AndroidLH上的表现分别优于先前代理32%、19%、15%和79%。
原文摘要 · Abstract (English)
Recent efforts to leverage the Multi-modal Large Language Model (MLLM) as GUI agents have yielded promising outcomes. However, these agents still struggle with long-horizon tasks in online environments, primarily due to insufficient knowledge and the inherent gap between offline and online domains. In this paper, inspired by how humans generalize knowledge in open-ended environments, we propose a Hierarchical Multimodal Skills (HMS) module to tackle the issue of insufficient knowledge. It progressively abstracts trajectories into execution skills, core skills, and ultimately meta-skills, providing a hierarchical knowledge structure for long-horizon task planning. To bridge the domain gap, we propose the Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm, which efficiently leverages skills acquired in offline environments to reduce the action search space during online tree exploration. Building on HMS, we propose Mirage-1, a multimodal, cross-platform, plug-and-play GUI agent. To validate the performance of Mirage-1 in real-world long-horizon scenarios, we constructed a new benchmark, AndroidLH. Experimental results show that Mirage-1 outperforms previous agents by 32\%, 19\%, 15\%, and 79\% on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH, respectively. Project page: https://cybertronagent.github.io/Mirage-1.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。