arXiv:2510.24694cs.CLcs.AI2025-10被引 7

用实体信息提升搜索代理训练,让错误答案也变得有用。

Repurposing Synthetic Data for Fine-grained Search Agent Supervision

  • 引入实体感知奖励机制,基于推理中识别的实体数量给分
  • 在多个问答和深度研究任务上,准确率显著高于基线方法
  • 能从接近正确的错误样本中学习,减少工具调用次数

基于大模型的搜索代理越来越多地使用以实体为中心的合成数据来解决复杂、知识密集型任务。然而,现有训练方法如组相对策略优化(GRPO)会丢弃这些丰富的实体信息,仅依赖稀疏的结果奖励。这一关键限制使模型无法区分有正确推理但最终答案错误的“近似正确”样本与完全失败的样本,从而丢失宝贵的训练信号。我们通过重新利用训练中被丢弃的实体信息来解决此问题。实证分析显示,代理推理过程中识别出的真实实体数量与最终答案准确率呈强正相关。基于此洞察,我们提出实体感知组相对策略优化(E-GRPO),构建一种密集的实体感知奖励函数,对错误样本按其实体匹配率给予部分奖励,使模型能有效从“近似正确”样本中学习。在多样化的问答(QA)和深度研究基准上的实验表明,E-GRPO持续且显著优于GRPO基线。此外,分析显示E-GRPO不仅准确率更高,还诱导出更高效的推理策略,所需工具调用次数更少,展现出更优的对齐效果和样本效率。

原文摘要 · Abstract (English)

LLM-based search agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training methods like Group Relative Policy Optimization (GRPO) discard this rich entity information, relying instead on sparse, outcome-based rewards. This critical limitation renders them unable to distinguish informative "near-miss" samples-those with substantially correct reasoning but a flawed final answer-from complete failures, thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals a strong positive correlation between the number of ground-truth entities identified during an agent's reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these "near-misses". Experiments on diverse question-answering (QA) and deep research benchmarks show that E-GRPO consistently and significantly outperforms the GRPO baseline. Furthermore, our analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient reasoning policies that require fewer tool calls, demonstrating a more effective and sample-efficient approach to aligning search agents.

搜索代理强化学习合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。