arXiv:2506.18670cs.IR2025-06KDD

用强化学习同时优化查询和文档,提升大模型检索效果。

Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation

  • 通过双向强化学习同步优化查询与文档的改写策略。
  • 在困难检索任务中显著提升性能,跨数据集泛化能力强。
  • 适合需要高精度信息检索的研究者和工程师使用。

近期研究提出利用大语言模型(LLM)通过重写查询来实现信息检索。然而,在复杂语料上,仅增强查询不足以实现稳健的语义匹配;大模型还需通过直接处理和增强文档本身来深入理解语料。为此,我们提出一种可同时对用户查询和语料文档进行增强的LLM检索器,并通过强化学习(RL)完全探索其策略,且极少依赖人工先验。值得注意的是,仅允许模型修改文档无法带来明显收益,除非搭配我们精心设计的双向强化学习框架,使模型能同时学习并协作优化查询与文档的增强策略。实现该框架的关键技术挑战在于训练过程中需联合更新两个策略,而两者的奖励相互依赖,导致奖励纠缠难以求解。我们的方法通过引入奖励采样策略和专门设计的强化学习算法,实现了基于采样奖励的有效训练。实验结果表明,该方法在稀疏和密集检索场景下均显著提升基于大模型的检索性能,尤其在困难检索领域表现突出,并展现出强大的跨基准泛化能力。代码已开源:https://github.com/liujm2001/CoAugRetriever。

原文摘要 · Abstract (English)

Recent studies have proposed leveraging Large Language Models (LLMs) as information retrievers through query rewriting. However, for challenging corpora, we argue that enhancing queries alone is insufficient for robust semantic matching; the LLM should also have sufficient understanding of the corpus by directly handling and augmenting the documents themselves. To this end, we present an LLM-based retriever empowered to augment both user queries and corpus documents, with its policy fully explored via reinforcement learning (RL) and minimal human inductive bias. Notably, we find that simply allowing the LLM to modify documents yields little benefit unless paired with our carefully designed bidirectional RL framework, which enables the LLM to simultaneously learn and collaborate on both query and document augmentation policies. A key technical challenge in realizing such a framework lies in jointly updating both policies during training, where the rewards for the two directions depend on each other, making their entangled reward intractable. Our approach addresses this by introducing a reward sampling strategy and a specifically designed RL algorithm that enables effective training with these sampled rewards. Experimental results demonstrate that our approach significantly enhances LLM-based retrieval performance in both sparse and dense settings, particularly in difficult retrieval domains, and achieves strong cross-benchmark generalization. Our code is released at https://github.com/liujm2001/CoAugRetriever.

大模型检索强化学习查询改写文档增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。