构建可扩展的图文混合交互环境，让计算机代理高效协同使用图形界面与命令行。
CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
- 将真实桌面软件转化为可复现的图文混合环境，自动生成命令行接口。
- 在多个基准上实现成功率提升39.3分，步骤减少37%，指令量降低60%。
- 适合研究智能代理跨模态协作、自动化办公系统开发人员参考。
计算机使用代理在OSWorld和AndroidWorld等基准上取得进展，但仍主要依赖图形界面，轨迹效率低。真实工作场景是图文混合的，需协调视觉状态与高吞吐命令行操作。现有环境因需为每款应用手动开发而难以扩展；代理也难互补：纯命令行代理缺乏视觉感知，纯图形代理执行命令效率低下。我们提出CUA-Universe，一个可扩展的环境-数据流水线，将真实桌面软件转为图文混合环境。App-Forge将应用转化为可复现虚拟机及命令行接口，支持16个应用；Task-Weave从种子文件合成可控难度的混合任务；Path-Steer引导高效混合路径并采集验证轨迹用于后训练。在该数据上训练的9B模型显著提升表现：CUA-Verse（得分+39.3，步骤-37%，令牌-60%），OSWorld（成功率+16.8，步骤-57%，令牌-44%），OSWorld-MCP（得分+7.84，步骤-27%，令牌-30%）。该框架为更强大高效的计算机使用代理提供可扩展路径。
原文摘要 · Abstract (English)
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。