用强化学习缓解界面智能体遗忘问题,提升持续学习能力。
CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning
- 结合监督微调与强化学习,动态调节两者权重以平衡效率与记忆保留。
- 在安卓控制持续学习基准上,性能超越基线模型37.2%以上。
- 适合研究智能体持续学习、人机交互系统开发的学者与工程师。
图形用户界面(GUI)智能体得益于多模态大语言模型的发展取得显著进展。然而,由于GUI应用频繁更新,如何在不遗忘旧任务的前提下适应新任务,仍是开放难题。本文发现,监督微调虽能快速适应,但易导致知识覆盖;而强化学习具有内在鲁棒性,可保护原有交互逻辑。基于此,我们提出持续GUI学习(CGL)框架,通过增强监督微调与强化学习的协同作用,动态平衡适应效率与技能保留。具体而言,设计基于策略熵的监督微调比例调节机制,动态控制两阶段权重分配;为解决显式梯度冲突,提出专用梯度手术策略,将探索性监督微调梯度投影到GRPO锚定梯度上,显式裁剪与GRPO冲突的梯度分量。此外,构建AndroidControl-CL基准,将GUI应用划分为不同任务组,有效模拟并评估持续学习表现。实验表明,所提框架在多种持续学习场景中均具优越性。相关基准、代码与模型将公开共享。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) Agents, benefiting from recent advances in multimodal large language models (MLLM), have achieved significant development. However, due to the frequent updates of GUI applications, adapting to new tasks without forgetting old tasks in GUI continual learning remains an open problem. In this work, we reveal that while Supervised Fine-Tuning (SFT) facilitates fast adaptation, it often triggers knowledge overwriting, whereas Reinforcement Learning (RL) demonstrates an inherent resilience that shields prior interaction logic from erasure. Based on this insight, we propose a \textbf{C}ontinual \textbf{G}UI \textbf{L}earning (CGL) framework that dynamically balances adaptation efficiency and skill retention by enhancing the synergy between SFT and RL. Specifically, we introduce an SFT proportion adjustment mechanism guided by policy entropy to dynamically control the weight allocation between the SFT and RL training phases. To resolve explicit gradient interference, we further develop a specialized gradient surgery strategy. By projecting exploratory SFT gradients onto GRPO-based anchor gradients, our method explicitly clips the components of SFT gradients that conflict with GRPO. On top of that, we establish an AndroidControl-CL benchmark, which divides GUI applications into distinct task groups to effectively simulate and evaluate the performance of continual GUI learning. Experimental results demonstrate the effectiveness of our proposed CGL framework across continual learning scenarios. The benchmark, code, and model will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。