arXiv:2602.09662cs.CV2026-02被引 3

用树结构高效生成可验证的GUI自动化轨迹,提升规划能力与泛化性。

TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution

  • 构建多智能体协作框架,以树形结构组织探索路径并复用节点。
  • 在多个任务上实现轨迹生成效率提升,跨域测试显示强泛化能力。
  • 适合研究自动化代理、GUI规划及大规模数据生成的开发者。

有效扩展GUI自动化对计算机使用代理(CUAs)至关重要;然而现有工作主要关注GUI定位的扩展,而非更关键的GUI规划,后者需要更复杂的数据收集。实际上,CUA在应用、桌面或网页间的探索过程通常呈树状结构,早期功能入口常被频繁访问。因此,将大规模轨迹组织为树结构可降低数据成本,优化GUI规划的数据扩展。本文提出TreeCUA,通过树结构化的可验证演化实现高效GUI自动化扩展。设计多智能体协作框架,用于环境探索、动作验证、轨迹总结与质量评估,生成高质量且可扩展的GUI轨迹。为提高效率,提出新型树状拓扑存储并重用重复探索节点,并设计自适应探索算法平衡轨迹深度(难度)与广度(多样性)。此外,引入世界知识引导与全局记忆回溯机制,避免低质量生成。最后,基于丰富的树节点信息自然扩展出TreeCUA-DPO方法,通过参考邻近轨迹分支信息提升GUI规划能力。实验结果表明,TreeCUA与TreeCUA-DPO均有显著提升,跨域(OOD)研究进一步证明其强泛化性。所有轨迹节点信息与代码将公开于https://github.com/UITron-hub/TreeCUA。

原文摘要 · Abstract (English)

Effectively scaling GUI automation is essential for computer-use agents (CUAs); however, existing work primarily focuses on scaling GUI grounding rather than the more crucial GUI planning, which requires more sophisticated data collection. In reality, the exploration process of a CUA across apps/desktops/web pages typically follows a tree structure, with earlier functional entry points often being explored more frequently. Thus, organizing large-scale trajectories into tree structures can reduce data cost and streamline the data scaling of GUI planning. In this work, we propose TreeCUA to efficiently scale GUI automation with tree-structured verifiable evolution. We propose a multi-agent collaborative framework to explore the environment, verify actions, summarize trajectories, and evaluate quality to generate high-quality and scalable GUI trajectories. To improve efficiency, we devise a novel tree-based topology to store and replay duplicate exploration nodes, and design an adaptive exploration algorithm to balance the depth (\emph{i.e.}, trajectory difficulty) and breadth (\emph{i.e.}, trajectory diversity). Moreover, we develop world knowledge guidance and global memory backtracking to avoid low-quality generation. Finally, we naturally extend and propose the TreeCUA-DPO method from abundant tree node information, improving GUI planning capability by referring to the branch information of adjacent trajectories. Experimental results show that TreeCUA and TreeCUA-DPO offer significant improvements, and out-of-domain (OOD) studies further demonstrate strong generalization. All trajectory node information and code will be available at https://github.com/UITron-hub/TreeCUA.

GUI自动化树结构多智能体规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。