arXiv:2503.18665cs.CV2025-03ICML被引 8

提出可分步评估动作的通用奖励模型,提升虚拟代理学习与推理能力。

Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark

  • 构建五维评估框架,分步生成多维度动作评分数据
  • 训练出的Similar模型在任务执行中表现优于基线,提升推理效率
  • 首个虚拟代理领域多维奖励模型基准,适合研究通用智能体的开发者

通用虚拟代理(GVAs)在自主任务执行方面展现出巨大潜力,但现有训练范式存在依赖结果监督和人工标注成本高等问题。为此,我们提出Similar——一种分步、多维度、通用型奖励模型,可为代理训练提供细粒度信号,并在推理时实现更优动作选择。首先,系统定义了评估代理行为的五个维度;基于此框架,设计MCTS-P算法自动收集并标注分步、五维的动作执行数据。利用该数据,采用三重策略(Triple-M)训练Similar模型。此外,我们构建了首个面向分步多维奖励模型的虚拟代理领域基准SRM,包含用于训练的SRMTrain和手动精选的测试集SRMEval。实验表明,Similar通过分步多维评估与协同增益,在训练及推理阶段均能有效提供中间信号,显著提升代理性能。项目代码已开源:https://github.com/antgroup/Similar。

原文摘要 · Abstract (English)

The development of Generalist Virtual Agents (GVAs) has shown significant promise in autonomous task execution. However, current training paradigms face critical limitations, including reliance on outcome supervision and labor-intensive human annotations. To address these challenges, we propose Similar, a Step-Wise Multi-Dimensional Generalist Reward Model, which offers fine-grained signals for agent training and can choose better action for inference-time scaling. Specifically, we begin by systematically defining five dimensions for evaluating agent actions. Building on this framework, we design an MCTS-P algorithm to automatically collect and annotate step-wise, five-dimensional agent execution data. Using this data, we train Similar with the Triple-M strategy. Furthermore, we introduce the first benchmark in the virtual agent domain for step-wise, multi-dimensional reward model training and evaluation, named SRM. This benchmark consists of two components: SRMTrain, which serves as the training set for Similar, and SRMEval, a manually selected test set for evaluating the reward model. Experimental results demonstrate that Similar, through its step-wise, multi-dimensional assessment and synergistic gain, provides GVAs with effective intermediate signals during both training and inference-time scaling. The project is available at https://github.com/antgroup/Similar.

虚拟代理奖励模型多维度评估强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。