提出自进化框架,让智能体自动优化图形界面与API交互策略。
EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning
- 将GUI与API交互统一为混合策略学习问题,动态选择最优模态。
- 在桌面应用测试中,分治策略使任务通过率提升17.8个百分点。
- 无需微调即可通过经验库实现推理时改进,适合多场景自动化。
结合图形界面操作与结构化API调用的计算机使用智能体,基于模型上下文协议(MCP)在自动化软件任务方面展现出潜力。然而,现有方法缺乏对智能体如何平衡两种模态的系统性理解,也难以在多样化应用中实现迭代自提升。本文将MCP-GUI的协同作用建模为统一的混合策略学习问题,使智能体学会在不同场景下判断哪种模态更具优势。研究发现,知识蒸馏与经验增强针对的是根本不同的失败模式,需根据应用特性选择机制。基于此,我们提出一个全自动自进化框架,集成环境自动生成与验证、轨迹采集、差距驱动的任务合成及质量过滤训练,全程无需人工干预。关键创新是经验银行,可积累大语言模型从轨迹对比中学习到的规则,在推理阶段实现性能提升而无需微调。跨三个桌面应用的系统分析表明,最优策略取决于MCP-GUI构成:在以MCP为主的任务中,蒸馏法达到77.8%通过率(+17.8pp);在以GUI为主的任务中,经验银行表现更优(+10.0pp)。
原文摘要 · Abstract (English)
Computer-use agents that combine GUI interaction with structured API calls via the Model Context Protocol (MCP) show promise for automating software tasks. However, existing approaches lack a principled understanding of how agents should balance these two modalities and how to enable iterative self-improvement across diverse applications. We formulate MCP-GUI interplay as a unified hybrid policy learning problem where the agent learns when each modality provides complementary advantages, and show that distillation and experience augmentation target fundamentally different failure modes - requiring application-aware mechanism selection. Built on this formulation, we propose a self-evolving framework with a fully automatic pipeline that orchestrates automatic environment generation and validation, trajectory collection, gap-driven task synthesis, and quality-filtered training - all without manual intervention. A key innovation is our experience bank, which accumulates LLM-learned rules from trajectory comparison, enabling inference-time improvement without fine-tuning. Systematic \textbf{cross-application analysis} across three desktop applications reveals that the optimal strategy depends on MCP-GUI composition: distillation achieves 77.8\% pass rate on MCP-dominant tasks (+17.8pp), while the experience bank excels on GUI-intensive tasks (+10.0pp).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。