arXiv:2506.18158cs.AIcs.CV2025-06被引 9

为跨应用导航的GUI智能体设计显式记忆机制,提升任务理解能力。

Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

  • 通过动作描述与屏幕信息构建显式记忆链,实现短期与长期记忆管理。
  • 在11.1万对屏幕-动作数据上验证,7B模型性能接近72B模型水平。
  • 适用于需要长程记忆的跨应用自动化场景,如智能助手、流程机器人。

多模态大语言模型(MLLM)正被广泛应用于图形用户界面(GUI)智能体开发。现有方法常依赖历史截图或操作隐式表示任务状态,难以准确理解复杂长任务中的上下文,缺乏有效机制存储关键信息。为此,我们提出链式记忆(Chain-of-Memory, CoM),一种显式建模GUI智能体短时与长时记忆的新方法。CoM通过捕获动作描述、融合任务相关屏幕信息,并配备专用记忆模块进行存储与管理。借助显式记忆表示,智能体能更精准理解任务状态并持久保留关键历史信息。为支持记忆管理能力训练与评估,我们构建了GUI Odyssey-CoM数据集,包含11.1万对标注有链式记忆的屏幕-动作样本。实验表明,CoM显著提升智能体在跨应用任务中的表现,且使7B模型的记忆能力达到72B模型水平。代码与数据集将开源。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) are attracting growing attention in the development of Graphical User Interface (GUI) agents. Existing approaches often rely on historical screenshots or actions to implicitly represent the task state. This reliance poses challenges for GUI agents in accurately understanding task states and underscores the absence of effective mechanisms to store critical information in complex and lengthy cross-app tasks. To address these challenges, we propose Chain-of-Memory (CoM), a novel approach for explicitly modeling short-term and long-term memory in GUI agents. CoM achieves this by capturing action descriptions, integrating task-relevant screen information, and maintaining a dedicated memory module to store and manage this information. By leveraging explicit memory representations, CoM enables GUI agents to better understand task states and retain critical historical information persistently. To equip GUI agents with memory management capabilities and evaluate the effectiveness of CoM, we developed the GUI Odyssey-CoM, a dataset comprising 111k screen-action pairs annotated with Chain-of-Memory. Experimental results demonstrate that CoM significantly improves GUI agents' performance in cross-application tasks. Additionally, GUI Odyssey-CoM enables 7B models to achieve memory management capabilities comparable to 72B models. The dataset and code will be open-sourced.

GUI智能体显式记忆跨应用导航MLLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。