构建高精度网页任务验证器,让AI代理的执行结果可信。
The Art of Building Verifiers for Computer Use Agents
- 用不重叠的评分标准和过程/结果分离设计,减少误判。
- 在长流程任务中接近人类一致率,假阳性率趋近于零。
- 适合想提升AI代理评估能力的研究者与开发者。
验证计算机使用代理(CUA)轨迹的成功与否是关键挑战:若验证不可靠,评估与训练信号均不可信。本文提出名为Universal Verifier的顶尖验证系统,基于四大原则:1)构建有意义且互不重叠的评分标准以降低噪声;2)分离过程与结果奖励,捕捉正确步骤但受阻或意外成功的情形;3)区分可控与不可控失败,通过级联无错策略实现更细粒度的失败分析;4)采用分治式上下文管理,关注轨迹中所有截图,提升长任务序列的可靠性。我们在新构建的CUAVerifierBench数据集上验证,该数据集包含人工标注的过程与结果标签,结果显示Universal Verifier与人类的一致性接近人类间一致性。相比基线模型WebVoyager(假阳性≥45%)和WebJudge(假阳性≥22%),本方法将假阳性率降至接近零。我们强调这些提升源于上述设计的累积效应。此外,自动研究代理仅用5%时间达到专家70%水平,但未能复现Universal Verifier所需全部策略。系统及数据集已开源,地址:https://github.com/microsoft/fara。
原文摘要 · Abstract (English)
Verifying the success of computer use agent (CUA) trajectories is a critical challenge: without reliable verification, neither evaluation nor training signal can be trusted. In this paper, we present lessons learned from building a best-in-class verifier for web tasks we call the Universal Verifier. We design the Universal Verifier around four key principles: 1) constructing rubrics with meaningful, non-overlapping criteria to reduce noise; 2) separating process and outcome rewards that yield complementary signals, capturing cases where an agent follows the right steps but gets blocked or succeeds through an unexpected path; 3) distinguishing between controllable and uncontrollable failures scored via a cascading-error-free strategy for finer-grained failure understanding; and 4) a divide-and-conquer context management scheme that attends to all screenshots in a trajectory, improving reliability on longer task horizons. We validate these findings on CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels, showing that our Universal Verifier agrees with humans as often as humans agree with each other. We report a reduction in false positive rates to near zero compared to baselines like WebVoyager ($\geq$ 45\%) and WebJudge ($\geq$ 22\%). We emphasize that these gains stem from the cumulative effect of the design choices above. We also find that an auto-research agent achieves 70\% of expert quality in 5\% of the time, but fails to discover all strategies required to replicate the Universal Verifier. We open-source our Universal Verifier system along with CUAVerifierBench; available at https://github.com/microsoft/fara.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。