用强化学习优化RAG检索，让模型自适应提升问答准确率。
Optimizing Retrieval for RAG via Reinforcement Learning
- 通过强化学习让检索器在RAG环境中自我探索优化
- 相比原检索器提升5.2%，超越现有最佳方法4.9%
- 仅需4张显卡一天训练，适合实际部署
随着检索增强生成（RAG）广泛应用，检索任务从辅助人类浏览转向支持AI推理，带来更复杂的搜索环境，传统相关性定义难以适应。现有检索器依赖人工标注或合成数据的监督微调，导致相关性固化，难以适配多样RAG场景。为此，我们提出R3：一种基于强化学习优化RAG的检索框架。该框架采用强化学习训练范式，使检索器在给定RAG环境中自主探索并持续改进，实现自动化学习，几乎无需人工调参。在多种任务上的大量实验表明，R3相比原始检索器提升5.2%性能，超越当前最优检索器4.9%，且达到使用微调大模型进行检索或RAG系统的水平。整个过程高效实用，仅需4张GPU，单日完成训练。
原文摘要 · Abstract (English)
As retrieval-augmented generation (RAG) becomes more widespread, the role of retrieval is shifting from retrieving information for human browsing to retrieving context for AI reasoning. This shift creates more complex search environments, where relevance is difficult to pre-define. Existing retrievers rely on supervised fine-tuning (SFT) with human labels or synthetic data, resulting in static relevance that struggles to adapt to diverse RAG environments. To address this challenge, we propose R3, a Retrieval framework optimized for RAG through Reinforcement learning (RL). Specifically, we adopt an RL training paradigm that enables the retriever to explore and self-improve within given RAG environments, automating the learning process with minimal manual experimentation or tuning effort. Extensive experiments across diverse tasks demonstrate that R3 improves RAG performance by 5.2% over the original retriever and surpasses state-of-the-art retrievers by 4.9%, while achieving comparable results to LLM-augmented retrieval and RAG systems built on post-trained or instruction-tuned LLMs. It is both efficient and practical, requiring only 4 GPUs and completing training within a single day.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。