用可执行记忆图提升GUI自动化效率与稳定性
Executable Agentic Memory for GUI Agent
- 构建结构化知识图谱，将界面操作转为检索执行
- 在AndroidWorld上性能领先基线19.6%，令牌消耗减少6倍
- 适合需要长流程自动化和低延迟的场景
现有GUI代理多采用以模型为中心的逐步交互方式，导致在长时序任务中易出错。本文提出可执行智能体记忆（EAM），一种结构化的知识图谱（KG），将GUI规划从自由生成转变为稳健的检索-执行过程。通过状态感知的深度优先搜索和动作分组挖掘，实现高效的记忆构建，压缩多步操作序列。为保障高效规划，引入轻量级Q函数引导蒙特卡洛树搜索（MCTS）在知识图谱上进行路径探索，并理论证明了Q模型的偏差一致性及路径恢复的样本复杂度。实验表明，EAM在AndroidWorld上相较UI-TARS-7B提升最高达19.6%，令牌开销仅为GPT-4o的1/6，平均延迟仅2.8秒，实现了可靠、快速且适用于长周期任务的GUI自动化。
原文摘要 · Abstract (English)
Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to $19.6\%$ on AndroidWorld, while reducing token costs $6\times$ relative to GPT-4o. With a $2.8$s average latency, EAM enables reliable, quick, and long-horizon GUI automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。