arXiv:2609.06530cs.CV2026-09

通过激活调控实现神经元级知识选择性保留,防止界面智能体遗忘旧应用知识。

Selective Knowledge Control for Continual GUI Agent Learning over Application Streams

论文配图:Selective Knowledge Control for Continual GUI Agent Learning over Application Streams
图 1 · 摘自论文原文
  • 基于前向激活动态控制梯度,区分共享与特定知识
  • 保护高激活神经元避免遗忘,截断未激活神经元防干扰
  • 实时更新历史知识状态,适合持续学习多应用场景

持续学习对图形用户界面(GUI)智能体适应不断演进的应用至关重要。应用流带来复杂的知识建模挑战:新应用常与旧应用共享底层知识,但也会引入需隔离的特定知识。本文提出激活条件下的选择性知识控制,一种轻量级方法,通过神经元级梯度操作实现选择性知识保留。该方法维护一个紧凑的历史知识状态,保护高度激活的MLP神经元以保留已有知识。当新应用到来时,根据前向激活执行实时梯度手术:将未激活神经元(持有特定知识)的梯度截断,防止干扰;将激活神经元(持有共享知识)的梯度正交投影,以保持稳定性并支持适应。每个应用阶段结束后,新识别的关键神经元被合并进历史状态用于后续学习。在多应用序列基准上的实验证明,该方法有效缓解对先前应用的灾难性遗忘,同时保持对新应用的鲁棒适应能力。

原文摘要 · Abstract (English)

Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.

持续学习知识保留GUI代理梯度手术

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。