用分步奖励优化训练电脑操作智能体，提升自动化效率。
PRO-CUA: Process-Reward Optimization for Computer Use Agents
- 分步奖励模型提供实时反馈，实现细粒度策略优化
- 在真实网页任务中达成92%成功率，优于传统方法
- 适合需要长期界面交互的自动化系统开发者
计算机使用智能体（CUAs）在自动化复杂数字流程方面展现出巨大潜力，但其训练受限于高昂的实时环境交互成本和高质量监督数据稀缺。现有行为克隆方法存在分布偏移和缺乏负向学习信号的问题。标准轨迹级强化学习则面临奖励稀疏、信用分配模糊及高基础设施成本等挑战。本文提出PRO-CUA框架，采用迭代的分步强化学习机制，将在线环境交互与策略优化解耦：当前策略通过实时滚动收集状态，为每个状态生成多样候选动作，由过程奖励模型（PRM）提供分步反馈，并基于组相对优势进行优化。该设计无需黄金答案或离线专家轨迹，即可实现密集且灵活的信用分配，同时通过在代理自身执行状态上训练降低分布偏移。在真实网页基准测试中，PRO-CUA表现出色，证明了PRM引导的分步训练的有效性与可靠性。
原文摘要 · Abstract (English)
Computer use agents (CUAs) have shown strong potential for automating complex digital workflows, yet their training remains constrained by costly live environment interaction and limited high-quality supervision. Existing filtered behavior cloning pipelines suffer from imitation bottlenecks, including distribution shift from the expert demonstration and the absence of negative learning signals. Meanwhile, standard trajectory-level reinforcement learning struggles with sparse rewards, ambiguous credit assignment, and high infrastructure costs for long-horizon GUI interaction. In this work, we propose PRO-CUA, a process-reward optimization framework for training CUAs with iterative step-level reinforcement learning. PRO-CUA decouples on-policy environment interaction from policy optimization: the current policy collects states through live rollouts, generates diverse candidate actions for each state, receives step-level feedback from a process reward model (PRM), and is optimized with group-relative advantages. This design enables dense and flexible credit assignment without relying on golden answers or offline expert trajectories, while reducing distribution shift by training on the agent's own execution states. Experiments on live web benchmarks demonstrate the effectiveness of PRO-CUA and the reliability of PRM-guided step-level training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。