arXiv:2512.16295cs.AI2025-12被引 6

构建跨平台GUI批评模型框架,提升自动化操作的可靠性。

OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

  • 自动生成跨平台GUI反馈数据,解决高质量训练数据稀缺问题。
  • 在310,000条样本上训练出性能领先的开源批评模型,移动端超越闭源模型。
  • 提供多平台评测基准,适合研究和部署自动化操作代理的开发者使用。

随着视觉语言模型驱动的计算机使用代理(CUAs)在图形用户界面(GUI)导航与操作方面能力增强,步骤级决策的可靠性已成为实际部署的关键瓶颈。在长时序任务中,错误快速累积,不可逆操作可能引发意外后果,促使需要在执行前评估每一步动作的批评模型。尽管批评模型具有前景,但其效果受限于缺乏多样且高质量的GUI反馈数据及公开的步骤级评估基准。为此,我们提出OS-Oracle框架,包含三项核心贡献:(1) 可扩展的跨平台GUI批评数据合成管道;(2) 结合监督微调(SFT)与一致性保持组相对策略优化(CP-GRPO)的两阶段训练范式;(3) 覆盖移动端、网页端和桌面端的综合性评测基准OS-Critic Bench。基于该框架,我们构建了包含31万条样本的高质量数据集。由此训练的OS-Oracle-7B模型,在开源VLM中达到当前最优水平,并在移动端表现超越闭源模型。此外,作为预批评模型,它显著提升了UI-TARS-1.5-7B等原生GUI代理在OSWorld和AndroidWorld环境中的性能。代码已开源。

原文摘要 · Abstract (English)

With VLM-powered computer-using agents (CUAs) becoming increasingly capable at graphical user interface (GUI) navigation and manipulation, reliable step-level decision-making has emerged as a key bottleneck for real-world deployment. In long-horizon workflows, errors accumulate quickly and irreversible actions can cause unintended consequences, motivating critic models that assess each action before execution. While critic models offer a promising solution, their effectiveness is hindered by the lack of diverse, high-quality GUI feedback data and public critic benchmarks for step-level evaluation in computer use. To bridge these gaps, we introduce OS-Oracle that makes three core contributions: (1) a scalable data pipeline for synthesizing cross-platform GUI critic data; (2) a two-stage training paradigm combining supervised fine-tuning (SFT) and consistency-preserving group relative policy optimization (CP-GRPO); (3) OS-Critic Bench, a holistic benchmark for evaluating critic model performance across Mobile, Web, and Desktop platforms. Leveraging this framework, we curate a high-quality dataset containing 310k critic samples. The resulting critic model, OS-Oracle-7B, achieves state-of-the-art performance among open-source VLMs on OS-Critic Bench, and surpasses proprietary models on the mobile domain. Furthermore, when serving as a pre-critic, OS-Oracle-7B improves the performance of native GUI agents such as UI-TARS-1.5-7B in OSWorld and AndroidWorld environments. The code is open-sourced at https://github.com/numbmelon/OS-Oracle.

GUI代理批评模型跨平台自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。