让GUI智能体学会总结经验,自动积累成功操作记忆。
EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration
- 用奖励模型自动收集成功操作轨迹,无需人工标注。
- 新任务时检索历史记忆,成功率提升显著。
- 适合需要长期优化的自动化工具开发者。
当前基于大视觉语言模型的GUI智能体虽能力增强,但往往将每个任务孤立处理,缺乏系统性学习过往经验的能力,导致重复出错、泛化能力差。为此,我们提出EchoTrail-GUI框架,模拟人类经验学习机制,赋予智能体动态可访问的记忆能力。该框架分三阶段运行:首先在经验探索阶段,智能体自主与GUI环境交互,由奖励模型验证并自动构建成功任务轨迹数据库,全程无需人工干预;其次在记忆注入阶段,针对新任务高效检索最相关的历史轨迹作为可行动记忆;最后在任务推理阶段,将这些记忆作为上下文引导,辅助智能体决策。在Android World和AndroidLab等基准测试中,结果表明,相较于基线模型,EchoTrail-GUI显著提升了任务成功率与执行效率,验证了结构化记忆对构建更鲁棒、智能的GUI自动化系统的有效性。
原文摘要 · Abstract (English)
Contemporary GUI agents, while increasingly capable due to advances in Large Vision-Language Models (VLMs), often operate with a critical limitation: they treat each task in isolation, lacking a mechanism to systematically learn from past successes. This digital ''amnesia'' results in sub-optimal performance, repeated errors, and poor generalization to novel challenges. To bridge this gap, we introduce EchoTrail-GUI, a novel framework designed to mimic human-like experiential learning by equipping agents with a dynamic, accessible memory. Our framework operates in three distinct stages. First, during Experience Exploration, an agent autonomously interacts with GUI environments to build a curated database of successful task trajectories, validated by a reward model. Crucially, the entire knowledge base construction is thus fully automated, requiring no human supervision. Second, in the Memory Injection stage, upon receiving a new task, our system efficiently retrieves the most relevant past trajectories to serve as actionable ''memories''. Finally, during GUI Task Inference, these memories are injected as in-context guidance to inform the agent's reasoning and decision-making process. We demonstrate the efficacy of our approach on benchmarks including Android World and AndroidLab. The results show that EchoTrail-GUI significantly improves the task success rate and operational efficiency of baseline agents, validating the power of structured memory in creating more robust and intelligent GUI automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。