arXiv:2605.09317cs.CLcs.CV2026-05被引 1

让网页操作智能体直接用隐状态记忆,比传统文本记忆更高效。

Mem-W: Latent Memory-Native GUI Agents

论文配图:Mem-W: Latent Memory-Native GUI Agents
图 1 · 摘自论文原文
  • 把历史轨迹和当前任务信息融合成连续隐状态,作为智能体的内在记忆。
  • 在多个导航任务上提升最多30.0分,显著优于现有方法。
  • 适合做长时程网页/移动端自动化操作的研究者和开发者。

GUI智能体正越来越多地操作网页、移动和桌面应用,成功控制依赖于将视觉、流程和任务级证据从当前屏幕延续至后续操作。然而,多数智能体仍将记忆视为外部可读的符号化记录:历史被总结、分类、检索并以文本或结构化形式重新编码后才被策略使用。这导致经验存储形式与现代GUI策略实际处理的隐空间序列之间存在不匹配。我们提出Mem-W,一种原生支持隐状态记忆的GUI智能体,将记忆视为连续上下文的一部分而非辅助符号框架。Mem-W通过共享的轨迹-隐压缩器,将历史轨迹(经验记忆)和会话内片段(工作记忆)融合为紧凑的记忆令牌,并将其与当前GUI观测及局部上下文编织成单一连续嵌入序列,使智能体能通过机器原生接口感知成功、失败和未完成进展。Mem-W采用自蒸馏和结果感知监督训练,保留决策相关状态,同时过滤无关记忆。在四个网页与移动导航基准测试中,Mem-W持续优于多种骨干网络和记忆增强基线,性能提升最高达+30.0,表明隐状态原生记忆可成为长时程GUI代理的可扩展基础。

原文摘要 · Abstract (English)

GUI agents are beginning to operate the web, mobile, and desktop as interactive worlds, where successful control depends on carrying forward visual, procedural, and task-level evidence beyond the fleeting present screen. Yet most agents still treat memory as an external, human-readable artifact: histories are summarized, categorized, retrieved, and reinserted as text or structured records before being encoded again by the policy. This creates a mismatch between the representational form in which experience is stored and the latent embedding sequence over which modern GUI policies actually act. We introduce Mem-W, a series of latent-memory-native GUI agents that treat memory as part of the agent's continuous context rather than as an auxiliary symbolic scaffold. Mem-W weaves both historical trajectories (as experiential memory) and in-session segments (as working memory) into compact memory tokens through a shared trajectory-to-latent compressor. These tokens are woven with the current GUI observation and local context into one continuous embedding sequence, allowing the agent to read successes, failures, and unfinished progress through the same machine-native interface. Mem-W is trained with self-distillation and outcome-aware supervision to preserve decision-relevant state while filtering memory toward evidence that truly supports task success. Across four web and mobile navigation benchmarks, Mem-W consistently improves diverse backbones and memory-enhanced baselines, with gains of up to $+30.0$, suggesting that latent-context-native memory can serve as a scalable foundation for long-horizon GUI agency.

GUI代理隐状态记忆长时程控制智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。