用贡献加权提升大模型搜索代理的决策能力
Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
- 引入每轮检索与推理的贡献评分，优化过程监督
- 在多个基准上性能提升5.0%~6.3%，效果更稳定
- 适合研究搜索代理、强化学习优化的学者参考
搜索代理通过让大语言模型（LLMs）访问预训练阶段未覆盖的实时与长尾信息，扩展了其静态知识边界。尽管强化学习广泛用于训练此类代理，现有方法仍面临关键挑战：过程监督常因价值估计不稳，而结果监督则因稀疏的轨迹级奖励导致信用分配困难。为此，我们提出贡献加权组相对策略优化（CW-GRPO），将过程监督融入组相对策略优化框架。不同于直接优化过程奖励，CW-GRPO使用大模型评判器评估每轮搜索中的检索效用与推理正确性，生成每轮贡献分数，并据此重标结果导向的优势值，实现细粒度信用分配且不牺牲优化稳定性。在多个知识密集型基准测试中，CW-GRPO相较于标准GRPO在Qwen3-8B上提升5.0%，在Qwen3-1.7B上提升6.3%，展现出更高效的搜索行为。额外分析表明，成功轨迹的贡献集中于特定轮次，为搜索代理任务提供了实证洞察。
原文摘要 · Abstract (English)
Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。