让智能助手分担记忆与计算,提升长流程界面操作成功率
UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

- 将任务执行与辅助功能分离,按需调用记忆检索和计算器工具
- 在MemGUI-Bench上超越7B级基线模型,安卓任务提升17.1%
- 适合需要长期依赖与复杂计算的自动化界面场景
基于多模态大模型的图形界面代理在复杂交互任务中表现优异,但在长周期场景下仍面临记忆衰退、进度混淆和数学幻觉等问题。为此,我们提出UI-Copilot框架,让图形界面代理专注执行,轻量级协作者负责记忆检索与数值计算。通过记忆解耦分离持久观察与临时上下文,并训练策略代理根据任务需求选择性调用协作者作为检索器或计算器。为有效学习工具调用,我们设计工具集成策略优化(TIPO),分别通过单轮预测优化工具选择,以及多轮滚动优化任务执行。实验表明,UI-Copilot-7B在挑战性数据集MemGUI-Bench上达到领先性能,显著优于GUI-Owl-7B与UI-TARS-1.5-7B等7B级模型;在AndroidWorld上相较基础Qwen模型实现17.1%的绝对提升,体现其在真实界面任务中的强泛化能力。
原文摘要 · Abstract (English)
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。