arXiv:2508.21767cs.CV2025-08被引 19

UItron让自动化操作手机电脑更智能,尤其擅长中文应用。

UItron: Foundational GUI Agent with Advanced Perception and Planning

  • 构建交互式环境并系统优化数据工程,提升GUI理解能力
  • 在中文主流应用上表现超越现有模型,正确率显著提升
  • 开源模型适合研究者与开发者推进GUI自动化落地

GUI代理旨在实现对移动设备和电脑的自动化操作,是迈向通用人工智能的重要一步。随着视觉语言模型(VLMs)的快速发展,其强大的视觉理解与任务规划能力加速了GUI代理的发展。然而,由于操作轨迹数据稀缺、交互基础设施不足以及基础模型初始能力有限,构建高效GUI代理仍具挑战。本文提出UItron,一个开源的面向自动GUI代理的基础模型,具备先进的GUI感知、定位与规划能力。该工作强调系统性数据工程与交互基础设施作为核心支撑。通过一系列数据工程策略提升训练效果,并构建连接移动端与PC端的交互环境。训练中采用监督微调处理多种GUI场景下的感知与规划任务,随后引入课程强化学习框架,支持在线环境中的复杂推理与探索。实验表明,UItron在GUI感知、定位与规划基准测试中表现优异。尤其在主流中文移动应用上展现出显著优势,因现有方案普遍缺乏中文适配能力。为此,团队手动收集了超过一百万步的操作轨迹,覆盖前100款最流行应用,并建立离线与在线评估环境。结果证明,UItron在中文应用场景中取得显著进展,推动GUI代理向真实应用更进一步。

原文摘要 · Abstract (English)

GUI agent aims to enable automated operations on Mobile/PC devices, which is an important task toward achieving artificial general intelligence. The rapid advancement of VLMs accelerates the development of GUI agents, owing to their powerful capabilities in visual understanding and task planning. However, building a GUI agent remains a challenging task due to the scarcity of operation trajectories, the availability of interactive infrastructure, and the limitation of initial capabilities in foundation models. In this work, we introduce UItron, an open-source foundational model for automatic GUI agents, featuring advanced GUI perception, grounding, and planning capabilities. UItron highlights the necessity of systemic data engineering and interactive infrastructure as foundational components for advancing GUI agent development. It not only systematically studies a series of data engineering strategies to enhance training effects, but also establishes an interactive environment connecting both Mobile and PC devices. In training, UItron adopts supervised finetuning over perception and planning tasks in various GUI scenarios, and then develop a curriculum reinforcement learning framework to enable complex reasoning and exploration for online environments. As a result, UItron achieves superior performance in benchmarks of GUI perception, grounding, and planning. In particular, UItron highlights the interaction proficiency with top-tier Chinese mobile APPs, as we identified a general lack of Chinese capabilities even in state-of-the-art solutions. To this end, we manually collect over one million steps of operation trajectories across the top 100 most popular apps, and build the offline and online agent evaluation environments. Experimental results demonstrate that UItron achieves significant progress in Chinese app scenarios, propelling GUI agents one step closer to real-world application.

GUI代理多模态自动化中文适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。