通过图模型实现智能体搜索中每一步的精准奖励分配。
Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling
- 将世界知识建模为潜在图，以节点距离衡量每步进展
- 在四个基准上提升搜索效率，显著优于传统方法
- 适合需要精细过程优化的复杂推理任务
在智能体搜索中，轨迹级结果奖励无法量化每一步的行为贡献，而现有步骤级奖励方法通常依赖昂贵的树采样。本文将世界知识视为潜在的世界图，每个信息检索任务看作在潜在任务图中的搜索过程，有效步骤应推动图向答案节点推进。基于此先验，提出图距离贡献奖励（GDCR），通过训练时构建的实体-关系（ER）图，评估新检索和新引用实体到答案节点的距离，赋予步骤级奖励。进一步提出步骤优势策略优化（SAPO），将GDCR转化为步骤级优势，并与轨迹级结果优势结合。在四个挑战性基准上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
In Agentic Search, trajectory-level outcome rewards fail to quantify the behavioral contributions of individual steps, while existing step-level reward methods typically rely on costly tree sampling. We view world knowledge as a latent world graph and each IS task as search within a latent task graph, where effective steps should make graph progress toward the answer node. Based on this prior, we propose Graph-Distance Contribution Reward (GDCR), a step-level process reward that scores newly-retrieved and newly-cited entities by their distance to the answer node in a training-time Entity-Relation (ER) graph. We further propose Step Advantage Policy Optimization (SAPO), which converts GDCR into step-level advantages and combines them with trajectory-level outcome advantages. Experiments on four challenging benchmarks validate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。