通过步骤级过滤生成高质量计算机操作数据,提升智能体训练效率。
WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering
- 对强模型的噪声轨迹进行步骤级筛选,保留正确操作
- 构建13.3K条轨迹、267K步的高质量数据集WebSTAR
- 推出轻量级奖励模型StepRM,适合大规模部署
计算机使用智能体(CUA)可操作真实数字界面,但因图形界面交互成本高及高质量轨迹数据稀缺,训练困难。现有数据集依赖人工示范,难以扩展。替代方案是利用强模型合成数据,但其推演过程噪声大,错误或次优动作占比高,直接模仿无效。为此,我们提出一种可扩展的数据合成流程,通过步骤级过滤机制逐步评估并保留正确动作,辅以推理增强提升规划能力。基于此流程,我们构建了WebSTAR数据集,包含13.3K条轨迹和267K个经过分级的、富含推理的步骤,源自OpenAI的计算机使用预览模型。我们在WebSTAR上微调Qwen-2.5-VL-Instruct模型(7B与32B)。在WebVoyager测试中,7B模型仅通过监督微调即超越开源最强模型UI-TARS-1.5-7B超过15%。进一步基于步骤级标注,我们构建WebSCORE数据集,并训练出由o4-mini蒸馏而来的7B多模态过程奖励模型StepRM,其评分质量媲美原模型,但部署效率显著更高。结果表明,步骤级过滤是实现可扩展CUA训练的关键原则,并提供了两个新数据集(WebSTAR、WebSCORE)与一个轻量级奖励模型(StepRM)作为实用工具,推动鲁棒高效CUA的发展。
原文摘要 · Abstract (English)
Computer use agents (CUAs) can operate real-world digital interfaces but remain difficult to train due to the high cost of graphical user interface (GUI) interaction and the scarcity of high-quality trajectory data. Existing datasets rely on human demonstrations, limiting scalability. A natural alternative is to synthesize data from strong CUAs, yet their rollouts are highly noisy, with incorrect or suboptimal actions consisting a large proportion of the steps, making naive imitation ineffective. To tackle this challenge, we introduce a scalable data synthesis pipeline that transforms noisy rollouts into reliable supervision without human annotation. The core idea is step-level filtering, which evaluates actions individually to retain only correct steps, complemented by reasoning augmentation for improved planning. Using this pipeline, we construct WebSTAR, a dataset of 13.3K trajectories and 267K graded, reasoning-rich steps synthesized from OpenAI's computer-use-preview model. We train Qwen-2.5-VL-Instruct models (7B and 32B) on WebSTAR. On WebVoyager, our 7B model surpasses SoTA open-source CUA model UI-TARS-1.5-7B by more than 15% with only supervised finetuning. Building on step-level grading, we further create WebSCORE, a dataset of graded step-level actions, and train StepRM, a 7B multimodal process reward model distilled from o4-mini, which matches its grading quality while being far more efficient to deploy at scale. Our results establish step-level filtering as a key principle for scalable CUA training and construct two new datasets (WebSTAR, WebSCORE) and a lightweight process reward model (StepRM) as practical tools to advance robust and efficient CUAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。