让电脑操作智能体学会在点击输入和调用工具间智能切换，提升任务完成效率。
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
- 通过合成轨迹与自举强化学习，训练智能体自主判断何时用图形界面操作、何时调用工具。
- 在OSWorld-MCP数据集上达到46.85%准确率，比基线提升约66%，优于纯界面操作设置3.9%。
- 无需真实工具轨迹采集，适合构建高效、可扩展的通用数字助手。
计算机使用智能体（CUAs）既能执行点击、输入等原子图形界面操作，也能调用文件管理等高阶工具操作，但这种混合动作空间常导致其难以决定何时继续界面操作或切换到工具，从而产生次优执行路径。这源于高质量交错式GUI-Tool轨迹稀缺、真实工具轨迹采集成本高且脆弱，以及缺乏轨迹级监督信号。本文提出ToolCUA，一种端到端智能体，通过分阶段训练学习最优的GUI-Tool路径选择。首先设计交错式GUI-Tool轨迹生成流水线，复用大量静态界面轨迹并合成有依据的工具库，实现无需人工工程或真实轨迹收集的多样化轨迹生成。其次采用工具自举的GUI强化微调，结合预热监督微调与单轮强化学习，优化关键切换点决策。最后在高保真环境里通过在线代理强化学习优化，以工具高效路径奖励引导合理工具使用与更短路径。在OSWorld-MCP上的实验表明，ToolCUA达到46.85%准确率，相对基线提升约66%，成为同规模模型的新基准。其性能还优于纯界面设置3.9%，验证了混合动作空间的有效性。结果表明，混合动作空间训练是构建真实世界数字智能体的可行方向。代码开源：https://x-plug.github.io/ToolCUA/
原文摘要 · Abstract (English)
Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: https://x-plug.github.io/ToolCUA/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。