打造跨平台通用图形界面智能体,实现手机电脑自主操作。
OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
- 构建合成数据管道与解耦训练框架,提升交互精度。
- 在ScreenSpot-V2达96.3%准确率,安卓任务成功率79.1%。
- 支持移动端中文环境与桌面系统,适合自动化办公场景。
图形界面(GUI)智能体有望使基础模型完成真实世界任务,革新人机交互并提升生产力。本文提出OmegaUse,一个支持手机与桌面跨平台自主执行任务的通用型GUI智能体。其成功依赖高质量数据与有效训练方法。为此,我们设计了精细的数据构建流程,结合开源数据集与新型自动化合成框架——融合自底向上探索与自顶向下分类引导生成,构建高保真合成数据;训练上采用两阶段策略:先通过监督微调建立基础交互语法,再用组相对策略优化提升空间定位与序列规划能力。为平衡计算效率与智能推理,采用混合专家(MoE)架构。为评估跨终端能力,引入OS-Nav基准套件,涵盖中文安卓环境(ChiM-Nav)与Ubuntu桌面常规交互(Ubu-Nav)。实验表明,OmegaUse在多个主流GUI基准上表现优异,在ScreenSpot-V2达到96.3%的SOTA得分,在AndroidControl实现79.1%的步骤成功率;在OS-Nav中,于ChiM-Nav取得74.24%的步骤成功率,Ubu-Nav平均成功率达55.9%。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。