让界面智能体像人一样记事、复盘，自动变强。
SE-GA: Memory-Augmented Self-Evolution for GUI Agents
- 用分层记忆库动态调取过往经验，支持长时间任务规划。
- 在线收集数据自进化，使模型在真实环境中成功率超89%。
- 适合做复杂交互任务的智能体开发，尤其动态环境表现突出。
自主图形用户界面（GUI）智能体在执行多步骤任务时，常受限于上下文窗口大小和静态策略，难以适应动态环境。本文提出自演进式GUI智能体（SE-GA），融合分层记忆结构与迭代自提升机制。核心是测试时记忆扩展（TTME），通过动态检索情景、语义与经验记忆，在推理时提供关键上下文以支持长期规划。为实现持续学习，引入记忆增强型自演化（MASE）训练流程，利用TTME收集的数据稳定并优化基础策略。在离线与在线基准上的广泛评估表明，SE-GA在ScreenSpot上达到89.0%的成功率，在挑战性数据集AndroidControl-High上达75.8%。AndroidWorld基准上的显著提升也验证了其在动态环境中的卓越泛化能力。开源代码：https://github.com/jinshilong-dev/SE-GA
原文摘要 · Abstract (English)
Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, this work proposes the Self-Evolving GUI Agent (SE-GA), a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism. At the core of our approach is Test-Time Memory Extension (TTME), which facilitates long-term planning by dynamically retrieving episodic, semantic, and experiential memories to provide salient contexts during inference. To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE), which is a training pipeline that adopts the data collected by TTME to stabilize and enhance the agent's foundational policy. Extensive evaluations across both offline and online benchmarks demonstrate SE-GA achieves state-of-the-art performance, reaching success rates of 89.0\% on ScreenSpot and 75.8\% on the challenging AndroidControl-High dataset. Furthermore, significant improvements on the AndroidWorld benchmark highlight the superior generalization to dynamic environments. Open source code: https://github.com/jinshilong-dev/SE-GA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。