UFO2让电脑自动化任务更稳定可靠，支持多应用协同执行。
UFO2: The Desktop AgentOS
- 采用主机代理+应用专用代理架构，结合原生API与统一界面操作层。
- 在20多个真实应用上测试，执行准确率显著高于现有系统。
- 适合需要稳定桌面自动化、支持多任务并行的开发者和用户。
近年来，基于多模态大语言模型的计算机使用代理（CUAs）通过自然语言实现了复杂桌面工作流的自动化。然而，多数现有CUAs仍停留在概念原型阶段，受限于浅层操作系统集成、脆弱的截图交互方式以及干扰性执行。本文提出UFO2，一个面向Windows桌面的多代理AgentOS，将CUAs提升为可落地的系统级自动化方案。UFO2包含中央主机代理负责任务分解与协调，以及一组具备原生API、领域知识和统一GUI-API操作层的应用专用代理，实现稳健的任务执行同时保持模块化与可扩展性。混合控制检测管道融合了Windows UI自动化（UIA）与视觉解析，支持多样化的界面风格。运行时效率通过推测性多动作规划提升，降低每步大模型开销。此外，画中画（PiP）界面在隔离虚拟桌面中运行自动化，使代理与用户可并行操作互不干扰。我们在超过20个真实Windows应用上评估UFO2，结果表明其在鲁棒性和执行准确性上显著优于先前方法。研究证明，深度操作系统集成是实现可靠、用户对齐桌面自动化的可扩展路径。
原文摘要 · Abstract (English)
Recent Computer-Using Agents (CUAs), powered by multimodal large language models (LLMs), offer a promising direction for automating complex desktop workflows through natural language. However, most existing CUAs remain conceptual prototypes, hindered by shallow OS integration, fragile screenshot-based interaction, and disruptive execution. We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs into practical, system-level automation. UFO2 features a centralized HostAgent for task decomposition and coordination, alongside a collection of application-specialized AppAgent equipped with native APIs, domain-specific knowledge, and a unified GUI--API action layer. This architecture enables robust task execution while preserving modularity and extensibility. A hybrid control detection pipeline fuses Windows UI Automation (UIA) with vision-based parsing to support diverse interface styles. Runtime efficiency is further enhanced through speculative multi-action planning, reducing per-step LLM overhead. Finally, a Picture-in-Picture (PiP) interface enables automation within an isolated virtual desktop, allowing agents and users to operate concurrently without interference. We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs. Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。