让手机助手自动完成跨应用任务,还能自我改进。
MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions
- 用多个专用代理+中央协调器,分工协作处理跨应用任务。
- 在真实环境测试中,准确率显著优于现有单/多代理方案。
- 适合想自动化复杂手机操作的用户或研究智能助手的开发者。
手机智能体可帮助用户自动化日常手机任务,已成为研究热点。然而,现有以流程为导向的智能体在处理跨应用指令时面临三大挑战:(1)任务关系复杂,(2)应用环境多样,(3)多步执行中的错误传播与信息丢失。受面向对象编程启发,我们提出自进化多智能体框架MobileSteward,集成多个面向应用的StaffAgent,由中央协调的StewardAgent统一调度。框架包含三个模块:(1)动态招募生成基于信息流的调度图,显式关联跨应用任务;(2)任务指派将任务分配给具备应用专长的StaffAgent,应对应用差异;(3)调整评估通过反馈提示或关键信息传递,缓解错误传播与信息丢失。为持续提升性能,我们设计基于记忆的自进化机制,总结成功执行经验。我们构建了首个真实环境下英文跨应用基准(CAPBench)用于评估。实验表明,MobileSteward在复杂跨应用指令处理上优于单/多智能体框架,展现出更强的适应性与鲁棒性。
原文摘要 · Abstract (English)
Mobile phone agents can assist people in automating daily tasks on their phones, which have emerged as a pivotal research spotlight. However, existing procedure-oriented agents struggle with cross-app instructions, due to the following challenges: (1) complex task relationships, (2) diverse app environment, and (3) error propagation and information loss in multi-step execution. Drawing inspiration from object-oriented programming principles, we recognize that object-oriented solutions is more suitable for cross-app instruction. To address these challenges, we propose a self-evolving multi-agent framework named MobileSteward, which integrates multiple app-oriented StaffAgents coordinated by a centralized StewardAgent. We design three specialized modules in MobileSteward: (1) Dynamic Recruitment generates a scheduling graph guided by information flow to explicitly associate tasks among apps. (2) Assigned Execution assigns the task to app-oriented StaffAgents, each equipped with app-specialized expertise to address the diversity between apps. (3) Adjusted Evaluation conducts evaluation to provide reflection tips or deliver key information, which alleviates error propagation and information loss during multi-step execution. To continuously improve the performance of MobileSteward, we develop a Memory-based Self-evolution mechanism, which summarizes the experience from successful execution, to improve the performance of MobileSteward. We establish the first English Cross-APP Benchmark (CAPBench) in the real-world environment to evaluate the agents' capabilities of solving complex cross-app instructions. Experimental results demonstrate that MobileSteward achieves the best performance compared to both single-agent and multi-agent frameworks, highlighting the superiority of MobileSteward in better handling user instructions with diverse complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。