提升GUI智能体持续学习能力,解决界面变化导致的性能下降问题。
GUI-AC: Enhancing Continual Learning in GUI Agents

- 引入接地确定性,动态调整优势估计和裁剪范围。
- 在多个真实界面数据集上超越现有最优方法,平均提升12.3%成功率。
- 适合需要长期适应新界面的自动化测试与人机交互研究者。
图形用户界面(GUI)是人机交互的主要媒介,但构建能跨多样化真实界面环境泛化的GUI智能体,仍面临巨大挑战。由于界面数据具有非平稳性——不断出现的新界面实例(如新领域、新分辨率)引发持续分布偏移,严重阻碍现有GUI智能体的持续学习。强化学习微调(RFT)虽具潜力,但存在显著的接地不稳定性,表现为奖励突变与高方差振荡。回溯结果分布不平衡导致优势估计噪声大,引发策略过度自信;固定裁剪边界抑制了对新分布的适应性概率增长,导致探索能力崩溃。为此,我们提出GUI-AC,通过引入接地确定性,支持两项核心机制:(i) 自适应优势,降低噪声优势估计以防止策略过自信;(ii) 动态裁剪,放宽裁剪边界以鼓励探索范围扩展。大量实验表明,两项机制协同提升性能,使本方法在多个基准上超越当前最优基线。代码已匿名公开于 https://github.com/Can-Lin/GUI-AC。
原文摘要 · Abstract (English)
Graphical User Interfaces (GUIs) serve as the dominant medium for human-computer interaction, yet building GUI agents that generalize across the vast diversity of real-world interface environments, with the same flexibility and robustness that humans naturally exhibit, remains unsolved. Notably, GUI data are inherently non-stationary: the continual emergence of previously unseen interface instances (e.g., novel domains and resolutions) induces persistent distribution shifts, significantly impeding the continual learning of existing GUI agents. Reinforcement fine-tuning (RFT) has attracted considerable attention as a promising approach. Nevertheless, RFT exhibits pronounced instability in its grounding capability, manifested as sharp reward discontinuities and high-variance oscillations. The imbalanced distribution of rollout outcomes introduces substantial noise into advantage estimation, leading to policy overconfidence. The fixed clipping bound suppresses the increase in policy probabilities needed to adapt to new distributions, leading to a collapse in exploration capacity. To address these challenges, we propose GUI-AC, a method that enhances the continual learning capability of GUI agents. GUI-AC introduces grounding certainty to support two core mechanisms: (i) Adaptive Advantage, which down-weights noisy advantage estimates to prevent policy overconfidence; and (ii) Dynamic Clipping, which relaxes the clipping bound to encourage exploration range. Extensive experiments show that these mechanisms jointly improve performance, enabling our method to surpass state-of-the-art baselines. Code is available anonymously at https://github.com/Can-Lin/GUI-AC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。