用历史操作记录构建记忆库,让手机自动化助手更懂真实应用。
MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation
- 将过去操作轨迹转为结构化页面记忆库,保留界面布局与功能上下文。
- 通过相似度检索记忆中的页面,补足大模型对真实应用的理解不足。
- 双大模型架构执行任务,支持进度跟踪,实测表现优于现有方法。
基于大语言模型(LLM)的自主代理在通过图形用户界面(GUI)自动化移动设备任务方面展现出巨大潜力。然而,面对复杂现实任务时仍存在挑战,主要源于LLM对真实移动应用知识的缺乏,导致规划无效甚至产生幻觉。为此,我们提出新型框架MapAgent,利用历史操作轨迹构建记忆以增强当前任务规划。首先,设计基于轨迹的记忆机制,将任务执行轨迹转化为可复用的结构化页面记忆数据库;每个页面被提取为紧凑且全面的快照,捕捉其界面布局与功能上下文。其次,引入粗到细的任务规划方法,根据相似性从记忆库中检索相关页面并注入LLM规划器,弥补对真实应用场景理解的不足,实现更明智、上下文感知的任务规划。最后,通过双大模型架构支持的任务执行器将计划转化为可执行动作,确保任务进度的有效追踪。真实场景实验表明,MapAgent性能显著优于现有方法。代码将开源,以支持进一步研究。
原文摘要 · Abstract (English)
The recent advancement of autonomous agents powered by Large Language Models (LLMs) has demonstrated significant potential for automating tasks on mobile devices through graphical user interfaces (GUIs). Despite initial progress, these agents still face challenges when handling complex real-world tasks. These challenges arise from a lack of knowledge about real-life mobile applications in LLM-based agents, which may lead to ineffective task planning and even cause hallucinations. To address these challenges, we propose a novel LLM-based agent framework called MapAgent that leverages memory constructed from historical trajectories to augment current task planning. Specifically, we first propose a trajectory-based memory mechanism that transforms task execution trajectories into a reusable and structured page-memory database. Each page within a trajectory is extracted as a compact yet comprehensive snapshot, capturing both its UI layout and functional context. Secondly, we introduce a coarse-to-fine task planning approach that retrieves relevant pages from the memory database based on similarity and injects them into the LLM planner to compensate for potential deficiencies in understanding real-world app scenarios, thereby achieving more informed and context-aware task planning. Finally, planned tasks are transformed into executable actions through a task executor supported by a dual-LLM architecture, ensuring effective tracking of task progress. Experimental results in real-world scenarios demonstrate that MapAgent achieves superior performance to existing methods. The code will be open-sourced to support further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。