arXiv:2609.04869cs.AI2026-09

让电脑操作智能体持续积累并优化使用技能,提升任务完成效率。

From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents

论文配图:From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents
图 1 · 摘自论文原文
  • 通过交互轨迹和反馈构建可持久化、可版本管理的技能库
  • 四类应用中平均得分提升5.7至18.6个百分点,且具领域依赖稳定性
  • 适合研究智能体长期学习与技能复用的开发者与研究人员

计算机使用智能体能在图形界面中执行日益复杂的任务,但其交互经验通常具有瞬时性:单次任务中获得的过程知识无法系统保留、优化并在后续任务中复用。现有技能库提供外部过程知识,但其相对于无技能智能体的增量价值,以及在重复交互下的长期动态特性仍缺乏充分刻画。本文提出一种在线技能演化框架,将交互轨迹与评估反馈转化为持久、可版本化的可复用操作库。每次迭代基于冻结的库快照执行,证据驱动的技能更新在后续迭代中生效,无需修改模型参数。我们在四个OSWorld应用领域中对比了完整演化库系统与配置相同的空库对照组,采用相同的动作生成和GUI对齐模块、任务集及迭代周期。经过五轮空库预热后,完整系统在所有四个领域运行中均达到更高后预热平均评估分数,平均差距为5.7至18.6个百分点,且表现出领域依赖的时间稳定性。在GIMP中,溯源分析揭示了跨任务起源边界的技能检索与修订波动现象,即多次接受的编辑未能恢复原始任务。这些发现表明,演化技能库可作为可审计、共享的过程记忆,提升固定计算机使用栈的表现,但其收益具有条件性,反复修订并不保证原始任务恢复。代码已开源:https://github.com/LongtaoHu/Skill-Evo4GUI。

原文摘要 · Abstract (English)

Computer-use agents can execute increasingly complex tasks in graphical interfaces, but their interaction experience is typically transient: procedural knowledge acquired from one rollout is not systematically retained, refined, and reused in later tasks. Existing skill libraries provide external procedural knowledge, yet their incremental value over the same agent operating without skills, as well as their longitudinal dynamics under repeated interaction, remain insufficiently characterized. We present an online skill-evolution framework that converts interaction trajectories and evaluator feedback into a persistent, versioned library of reusable procedures. Each iteration executes against a frozen library snapshot, and evidence-guided skill updates become available in subsequent iterations without changing model parameters. We compare the full evolving-library system with a configuration-matched empty-library control across four OSWorld application domains under the same fixed action-generation and GUI-grounding stack, task sets, and iteration horizons. Following a five-iteration empty-library warm-up, Full attains a higher post-warm-up mean evaluator score in all four observed domain runs, with mean differences ranging from 5.7 to 18.6 percentage points and domain-dependent temporal stability. In GIMP, provenance-aware analysis reveals retrieval across task-of-origin boundaries and revision churn, where repeated accepted edits fail to recover the originating task. These findings characterize evolving skill libraries as auditable, shared procedural memory that can improve a fixed computer-use stack, while showing that their benefits are conditional and repeated revision does not guarantee recovery. Code is released at https://github.com/LongtaoHu/Skill-Evo4GUI.

智能体学习技能演化人机交互持续学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。