用强化学习训练嵌入模型，实现高效多步检索
Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Training
- 用强化学习优化嵌入模型，支持多步检索
- 在1000万词长上下文上达到顶尖性能
- 无需微调大模型，资源消耗更低
检索增强生成（RAG）通过高效筛选相关上下文提升大模型表现，减少幻觉并降低推理成本。但现有方法多为单步检索，难以应对需多步搜索的复杂问题。近期多步检索方法虽出现，通常依赖微调小型大模型，资源消耗高且无法使用更大模型。本文提出Q-RAG，通过强化学习微调嵌入模型实现多步检索，为开放域问答提供高效、资源友好的替代方案，在长达1000万词的上下文基准BabiLong和RULER上取得当前最优结果。代码已公开于https://github.com/griver/Q-RAG。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) methods enhance LLM performance by efficiently filtering relevant context for LLMs, reducing hallucinations and inference cost. However, most existing RAG methods focus on single-step retrieval, which is often insufficient for answering complex questions that require multi-step search. Recently, multi-step retrieval approaches have emerged, typically involving the fine-tuning of small LLMs to perform multi-step retrieval. This type of fine-tuning is highly resource-intensive and does not enable the use of larger LLMs. In this work, we propose Q-RAG, a novel approach that fine-tunes the Embedder model for multi-step retrieval using reinforcement learning (RL). Q-RAG offers a competitive, resource-efficient alternative to existing multi-step retrieval methods for open-domain question answering and achieves state-of-the-art results on the popular long-context benchmarks BabiLong and RULER for contexts up to 10M tokens. Code is available at https://github.com/griver/Q-RAG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。