arXiv:2608.04530cs.CV2026-08

将潜空间记忆拆解为内容、读取和信任三部分,提升GUI代理的记事与决策能力。

FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory

论文配图:FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory
图 1 · 摘自论文原文
  • 分角色设计内容基、状态感知读取与轻量信任门,分工管理记忆
  • 在五个基准上超越固定记忆基线,信任门可降低无关记忆干扰
  • 适合需要长期记忆与精准决策的自动化交互任务

GUI代理需同时记住过往经验与当前任务进展。潜空间记忆通过压缩多模态轨迹为连续向量实现紧凑存储,但现有方法通常将每条轨迹映射到单一固定记忆块,并主要依赖下一步动作监督训练,导致三个问题:压缩中丢失重要细节,同一记忆块需适应不同决策阶段,以及无关检索可能误导代理。本文提出FocusMem,将潜空间记忆接口中的职责解耦:角色感知的内容基促进回放记忆保留可复用经验,工作记忆保留任务进展;状态条件读取生成针对当前决策的证据视图;轻量信任门可抑制与当前步骤无关的记忆块。所有组件在保持GUI策略冻结的情况下联合训练。在五个GUI代理基准上,FocusMem持续优于仅动作监督的固定记忆基线及先前潜空间记忆改进方案。进一步分析表明,语义与功能监督分别保存互补信息,状态条件读取随上下文轨迹增长更鲁棒,信任门显著降低注入无关回放证据的损害。结果表明,有效潜空间记忆不仅依赖历史压缩,更取决于保留什么、暴露什么、允许什么。

原文摘要 · Abstract (English)

GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map each trajectory to one fixed memory block and train it mainly through next-action supervision. This creates three practical problems: important details may be lost during compression, the same memory block must serve different decision stages, and irrelevant retrieved trajectories may still mislead the agent. We introduce FocusMem, which separates these responsibilities within a compact latent-memory interface. A role-aware content basis encourages episodic memory to retain reusable experience and working memory to retain task progress. A state-conditioned readout generates a decision-specific view of the same stored evidence, while a lightweight trust gate can suppress memory blocks that appear irrelevant to the current step. All components are trained while the GUI policy remains frozen. Across five GUI-agent benchmarks, FocusMem consistently outperforms a fully matched action-only fixed-memory baseline and prior latent memory adaptations. Further analysis shows that semantic and functional supervision preserve complementary information, state-conditioned readout is more robust as surrounding trajectory context grows, and the trust gate reduces the harm caused by injected irrelevant episodic evidence. These results show that effective latent memory depends not only on compressing past interaction, but also on what is retained, what is exposed, and what is allowed.

GUI代理潜空间记忆注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。