通过分步导航与操作提升移动端智能体的任务完成能力
ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
- 将任务拆解为页面跳转和操作两阶段，避免局部最优
- 在步骤级和任务级分别提升7.12%、4.72%的界面匹配准确率
- 适合需要精准交互的自动化移动端应用开发场景
近期，移动端AI代理受到越来越多关注。给定一项任务，移动端AI代理可通过多步交互与移动设备互动，最终形成解决任务的GUI流程。然而，现有代理通常在每一步只关注最相关元素，导致陷入局部最优并忽略整体GUI流程。为此，我们构建了一个名为MobileReach的训练数据集，将任务分解为页面跳转和操作两个子任务。此外，我们提出了ReachAgent，一种两阶段框架，专注于提升任务完成能力。该框架结合页面跳转与操作子任务，并利用基于奖励的偏好GUI流程进一步优化。实验结果表明，相比当前最优（SOTA）代理，ReachAgent在步骤级上将IoU准确率和文本准确率分别提升了7.12%和7.69%，在任务级上分别提升了4.72%和4.63%。数据与代码将在论文接受后公开。
原文摘要 · Abstract (English)
Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the IoU Acc and Text Acc by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent. Our data and code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。