用信息增益优化搜索步骤,让大模型更精准地提问。
IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

- 每一步搜索都计算信息增益,判断检索结果是否提升答案信心。
- 在7个问答数据集上平均准确率43.0%,比最强基线高1.6点。
- 无需额外标注,适合多跳推理任务和资源有限的场景。
强化学习已成为训练大语言模型进行搜索增强推理的有效范式。然而,现有方法依赖轨迹级奖励,无法区分单次推演中不同搜索查询的优劣,且当所有采样轨迹均失败时梯度信号接近于零。本文提出IG-Search,一种基于信息增益(IG)的步级奖励强化学习框架。对于每一步搜索,IG衡量检索文档相较于随机文档基线使模型对正确答案置信度提升的程度,反映搜索查询的有效性。该信号通过GRPO中的逐令牌优势调节反馈至对应查询词元,实现推演内的细粒度信用分配。与以往需外部标注或共享环境状态的步级方法不同,IG-Search仅利用策略自身生成概率,除标准问答对外无需中间标注。在七个单跳与多跳问答基准上的实验表明,IG-Search使用Qwen2.5-3B模型实现平均精确匹配(EM)为0.430,优于最强轨迹级基线MR-Search的1.6点,以及步级方法GiGPO的0.9点,尤其在多跳推理任务中表现显著。尽管引入密集步级信号,其每步训练耗时仅比轨迹级基线增加约6.4%,推理延迟不变,且在所有采样轨迹错误时仍提供有效梯度信号。
原文摘要 · Abstract (English)
Reinforcement learning has emerged as an effective paradigm for training large language models to perform search-augmented reasoning. However, existing approaches rely on trajectory-level rewards that cannot distinguish precise search queries from vague or redundant ones within a rollout group, and collapse to a near-zero gradient signal whenever every sampled trajectory fails. In this paper, we propose IG-Search, a reinforcement learning framework that introduces a step-level reward based on Information Gain (IG). For each search step, IG measures how much the retrieved documents improve the model's confidence in the gold answer relative to a counterfactual baseline of random documents, thereby reflecting the effectiveness of the underlying search query. This signal is fed back to the corresponding search-query tokens via per-token advantage modulation in GRPO, enabling fine-grained, step-level credit assignment within a rollout. Unlike prior step-level methods that require either externally annotated intermediate supervision or shared environment states across trajectories, IG-Search derives its signals from the policy's own generation probabilities, requiring no intermediate annotations beyond standard question-answer pairs. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate that IG-Search achieves an average EM of 0.430 with Qwen2.5-3B, outperforming the strongest trajectory-level baseline (MR-Search) by 1.6 points and the step-level method GiGPO by 0.9 points on average across benchmarks, with particularly pronounced gains on multi-hop reasoning tasks. Despite introducing a dense step-level signal, IG-Search adds only ~6.4% to per-step training wall-clock time over the trajectory-level baseline and leaves inference latency unchanged, while still providing a meaningful gradient signal even when every sampled trajectory answers incorrectly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。