arXiv:2604.11784cs.LGcs.AI2026-04被引 14

打造统一框架,让GUI智能体可训练、可评估、可部署。

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

  • 构建端到端的GUI代理训练与部署系统
  • 移动端纯视觉任务成功率提升至17.1%
  • 支持多平台落地,适合跨设备智能体研发

GUI代理通过可视化界面而非程序接口操作应用,以点击、滑动和输入方式与任意软件交互,覆盖了命令行代理难以触及的长尾应用。然而该领域进展受限于缺乏统一的全栈基础设施:在线强化学习受环境不稳和封闭流程制约,评估标准在不同研究间漂移,训练好的代理极少能真正部署到真实设备上。本文提出开源框架ClawGUI,解决三大痛点。ClawGUI-RL首次提供开源的GUI代理强化学习基础设施,支持并行虚拟环境与真实物理设备,结合GiGPO与进程奖励模型实现密集步骤级监督。ClawGUI-Eval在6个基准和11+模型上推行标准化评估流程,复现率达95.8%。ClawGUI-Agent通过12+聊天平台将代理部署至Android、HarmonyOS和iOS,支持混合命令行与图形界面控制及持久个性化记忆。端到端训练的ClawGUI-2B在MobileWorld GUI-Only任务中取得17.1%的成功率,优于同规模基线MAI-UI-2B的6.0个百分点。

原文摘要 · Abstract (English)

GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present \textbf{ClawGUI}, an open-source framework addressing these three gaps within a single harness. \textbf{ClawGUI-RL} provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. \textbf{ClawGUI-Eval} enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8\% reproduction against official baselines. \textbf{ClawGUI-Agent} brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, \textbf{ClawGUI-2B} achieves 17.1\% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0\%.

GUI代理强化学习多平台部署自动化测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。