arXiv:2501.11733cs.CLcs.CV2025-01被引 134

让手机助手能像人一样从经验中自我进化,完成复杂多步骤任务。

Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks

  • 分层架构:高层规划与底层执行分离,提升任务分解能力。
  • 自进化机制:通过经验积累的技巧和快捷操作序列,性能提升22%。
  • 适合研究者与开发者:想构建能持续学习的智能手机助手可参考。

智能手机已成为现代生活不可或缺的工具,但复杂任务的操作仍令人困扰。尽管基于大模型的移动端智能体已具备环境感知与执行能力,现有方法仍难以满足真实需求,面对需要深度推理与长时程的任务表现不足,且缺乏从过往经验中学习与改进的能力。为此,我们提出 Mobile-Agent-E,一种分层多智能体框架,支持通过历史经验实现自我演化。该框架包含负责整体规划的 Manager 和四个子代理:负责视觉感知的 Perceptor、执行具体动作的 Operator、验证错误的 Action Reflector 及聚合信息的 Notetaker。系统还引入新型自演化模块,维护持久的长期记忆,包括通用经验(Tips)与可复用操作序列(Shortcuts),从而持续优化执行效率与准确率。同时,我们构建了 Mobile-Eval-E 基准,涵盖需跨应用、长周期交互的复杂任务。实验表明,Mobile-Agent-E 在三种基础模型下相比当前最优方法,绝对性能提升22%。

原文摘要 · Abstract (English)

Smartphones have become indispensable in modern life, yet navigating complex tasks on mobile devices often remains frustrating. Recent advancements in large multimodal model (LMM)-based mobile agents have demonstrated the ability to perceive and act in mobile environments. However, current approaches face significant limitations: they fall short in addressing real-world human needs, struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms to learn and improve from prior experiences. To overcome these challenges, we introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of self-evolution through past experience. By hierarchical, we mean an explicit separation of high-level planning and low-level action execution. The framework comprises a Manager, responsible for devising overall plans by breaking down complex tasks into subgoals, and four subordinate agents--Perceptor, Operator, Action Reflector, and Notetaker--which handle fine-grained visual perception, immediate action execution, error verification, and information aggregation, respectively. Mobile-Agent-E also features a novel self-evolution module which maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are general guidance and lessons learned from prior tasks on how to effectively interact with the environment. Shortcuts are reusable, executable sequences of atomic operations tailored for specific subroutines. The inclusion of Tips and Shortcuts facilitates continuous refinement in performance and efficiency. Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring complex mobile tasks requiring long-horizon, multi-app interactions. Empirical results show that Mobile-Agent-E achieves a 22% absolute improvement over previous state-of-the-art approaches across three foundation model backbones. Project page: https://x-plug.github.io/MobileAgent.

智能助手自进化移动智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。