为智能体推理过程设计结构化奖励模型，提升复杂任务表现
Exploring Reasoning Reward Model for Agents
- 构建多维度反馈机制，包含推理路径、问题诊断和综合评分
- 统一反馈集成策略在12个基准上实现43.7%（GAIA）与46.2%（WebWalkerQA）准确率
- 适用于需要高质量推理的智能体系统研究者
智能体强化学习在复杂推理与工具使用方面取得显著进展，但多数方法仍依赖稀疏的结果奖励。此类反馈无法区分中间推理质量，导致训练效果不佳。本文提出智能体推理奖励模型（Agent-RRM），提供三类结构化反馈：(1) 显式的推理轨迹，(2) 聚焦的批评意见以指出推理缺陷并提供优化建议，(3) 综合性能评分。基于这些信号，系统评估三种整合策略：Reagent-C（文本增强优化）、Reagent-R（奖励增强引导）和 Reagent-U（统一反馈融合）。在12个多样化基准上的实验证明，Reagent-U显著提升性能，在GAIA上达到43.7%，在WebWalkerQA上达到46.2%，验证了该推理奖励模型与训练方案的有效性。代码、模型与数据集均已开源，以促进后续研究。
原文摘要 · Abstract (English)
Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。