让电脑操作机器人记住关键画面和文字,长期任务更靠谱。
MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

- 用智能记忆控制器筛选、压缩、检索操作记录,只存有用信息。
- 在多个测试集上表现优于不记事、全屏重放和纯文字记忆方法。
- 适合需要长时间连续操作的自动化工具开发者使用。
近期的图形界面(GUI)智能体在视觉定位和动作预测方面取得显著进展,但在需跨多步骤保持任务状态的长周期任务中仍显脆弱。现有方法通常依赖原始历史重放或仅文本记忆,前者导致模型被冗余截图淹没,后者则丢失对后续决策至关重要的局部视觉证据。为此,我们提出 extbf{MementoGUI},一个可插拔的智能体记忆框架,为基于多模态大语言模型(MLLM)的GUI智能体配备 extbf{MementoCore},即一个用于在线记忆选择、压缩与检索的可学习控制器。MementoGUI 将长周期GUI控制建模为在线记忆控制问题:工作记忆有选择地保留任务相关的界面事件,附带文字摘要和感兴趣区域(ROI)级视觉证据;情景记忆通过学习相关性选择,检索可复用的过往轨迹。MementoCore 将记忆控制模块化为步处理、记忆压缩、情景写入和情景选择四个专用算子,实现无需微调主干模型的插件式记忆增强。我们进一步构建了可扩展的数据整理流程,将计算机使用轨迹转化为记忆控制器训练数据,提出 extbf{MementoGUI-Bench} 评估长周期决策能力,并设计基于MLLM的语义动作匹配、任务进度和记忆一致性指标。在 GUI-Odyssey、MM-Mind2Web 和 MementoGUI-Bench 上的实验表明,MementoGUI 持续优于无历史、历史重放和纯文本记忆基线,且更大规模的 MementoCore 进一步提升记忆增强型GUI控制性能。
原文摘要 · Abstract (English)
Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce \textbf{MementoGUI}, a plug-in agentic memory framework that equips MLLM-based GUI agents with \textbf{MementoCore}, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce \textbf{MementoGUI-Bench} for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。