arXiv:2606.31612cs.CV2026-06被引 3

提出主动记忆机制,让GUI智能体更准确判断任务状态。

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

论文配图:What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States
图 1 · 摘自论文原文
  • 将记忆从被动存储变为动态维护的任务状态
  • 引入在线强化学习,按需选择使用记忆提升成功率
  • 新评测基准可区分任务完成与错误操作

移动GUI代理面临需跨页面和应用读取、更新、复用任务数据的长时任务。现有方法将记忆视为被动存储,仅累积历史观测,导致在长轨迹中因重复值、干扰项和过期状态而难以判断数据是否应被使用、已使用或需等待。为此,我们提出主动任务驱动记忆(ATMem),将任务相关信息作为持续更新的执行状态维护,使每个值与其角色和当前状态关联,支持基于工作流状态的动作选择。虽然监督微调可构建ATMem,但无法教会何时使用。因此我们引入STR-GRPO,一种在线强化学习方法,通过对比开启/关闭记忆的回溯,估计记忆使用对任务完成的贡献,并通过考虑内存成本的奖励抑制无效使用。为评估代理是否能完成所有在范围内的任务且避免越界操作,我们构建了一个挑战性移动端基准。该基准要求从一组近似相同的条目中,对满足指令的条目进行操作并拒绝违反约束的条目。我们进一步引入应用级进展和范围感知F1,分别衡量这两个维度。

原文摘要 · Abstract (English)

Mobile GUI agents increasingly face long-horizon tasks that require reading, updating, and reusing task-relevant data across pages and applications. Existing methods treat memory largely as passive storage, where past observations are accumulated and retrieved when needed. Yet retrieving a value does not reveal its current role in the workflow. The agent must still infer from accumulated records whether the value should be used now, has already been used, or must wait for a later dependency. This implicit reconstruction becomes unreliable in long trajectories with repeated values, distractors, and outdated states, causing repeated or missed operations. To address this, we propose Active Task Driving Memory (ATMem), which shifts GUI-agent memory from passive storage to an actively maintained execution state. ATMem maintains task-relevant information as a continually updated execution state that links each value to its role and current status, enabling action selection based on the current workflow state. While supervised fine-tuning enables the agent to construct ATMem, it does not teach when ATMem is beneficial. We therefore introduce STR-GRPO, an online reinforcement learning method that encourages selective use of ATMem based on its contribution to task completion. STR-GRPO contrasts memory-on and memory-off rollouts to estimate when memory use improves execution, while memory-cost-aware reward discourages costly memory usage that does not improve execution. To evaluate whether agents can complete all in-scope work while avoiding out-of-scope actions, we build a challenging mobile benchmark. From a list of near identical entries, agents must act on every entry that satisfies the instruction and reject entries that violate its constraints. We further introduce App-Level Progress and Scope-Aware F1 to measure these two dimensions separately.

GUI代理主动记忆强化学习任务状态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。