让手机智能助手学会自动优化重复操作,又快又准。
AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
- 用记忆记录操作历史,自动发现可合并的重复动作
- 将重复步骤升级为快捷指令,提速40%以上且准确率不降
- 适合需要高效执行日常任务的AI助手开发者
大型语言模型(LLMs)推动了能够与图形用户界面(GUI)交互的智能代理发展。这些代理具备强大推理和适应能力,可执行传统需预设规则的复杂任务。但基于逐步推理的代理在处理常规任务时效率低下;而传统规则系统虽高效,却缺乏灵活性。为此,我们提出一种新型进化框架,通过记忆机制记录任务执行历史,分析重复操作序列,生成高层快捷动作替代低层操作,提升效率的同时保持智能与灵活性。实验在多个基准任务上显示,该方法在效率和准确率上均显著优于现有方法。代码将开源以支持后续研究。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have led to the development of intelligent LLM-based agents capable of interacting with graphical user interfaces (GUIs). These agents demonstrate strong reasoning and adaptability, enabling them to perform complex tasks that traditionally required predefined rules. However, the reliance on step-by-step reasoning in LLM-based agents often results in inefficiencies, particularly for routine tasks. In contrast, traditional rule-based systems excel in efficiency but lack the intelligence and flexibility to adapt to novel scenarios. To address this challenge, we propose a novel evolutionary framework for GUI agents that enhances operational efficiency while retaining intelligence and flexibility. Our approach incorporates a memory mechanism that records the agent's task execution history. By analyzing this history, the agent identifies repetitive action sequences and evolves high-level actions that act as shortcuts, replacing these low-level operations and improving efficiency. This allows the agent to focus on tasks requiring more complex reasoning, while simplifying routine actions. Experimental results on multiple benchmark tasks demonstrate that our approach significantly outperforms existing methods in both efficiency and accuracy. The code will be open-sourced to support further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。