让大模型反馈指导文档重排,动态调整检索数量与顺序。
DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
- 用大模型输出质量作奖励信号,通过强化学习优化重排策略。
- 在7个数据集上表现优于同规模模型,提升生成质量和可解释性。
- 适合需要精准知识检索的问答、写作等应用,尤其关注效率与准确率平衡。
检索增强生成(RAG)系统结合大语言模型(LLM)与外部知识检索,在知识密集型任务中表现优异。其中重排器是关键但常被忽视的组件:无关文档会误导生成结果,因此重排器需精炼检索结果以提升生成质量与可解释性。然而,确定重排应选取的文档数量(k)极具挑战——过少可能遗漏关键信息,过多则引入噪声并降低效率。尽管已有研究探索基于大模型的重排器,但主要依赖模型内部知识,忽略了大模型提供的丰富监督信号,例如利用生成质量作为反馈来优化重排决策。本文提出 DynamicRAG,一种新型 RAG 框架,其重排器可根据查询动态调整检索文档的顺序与数量。我们将重排器建模为一个通过强化学习优化的智能体,使用来自大模型输出质量的奖励信号。在七个知识密集型数据集上,DynamicRAG 在同参数规模模型中达到最先进性能。模型、数据与代码已开源:https://github.com/GasolSun36/DynamicRAG。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) systems combine large language models (LLMs) with external knowledge retrieval, making them highly effective for knowledge-intensive tasks. A crucial but often under-explored component of these systems is the reranker. Since irrelevant documents in RAG systems can mislead the generator, the reranker plays a vital role in refining retrieved documents to enhance generation quality and explainability. However, it is challenging to determine the appropriate number of documents ($k$) that the reranker should select: too few may result in missing critical information, while too many introduce noise and inefficiencies. Although recent studies have explored LLM-based rerankers, they primarily leverage internal model knowledge and overlook the rich supervisory signals that LLMs can provide, such as using response quality as feedback for optimizing reranking decisions. In this paper, we propose DynamicRAG, a novel RAG framework where the reranker dynamically adjusts both the order and number of retrieved documents based on the query. We model the reranker as an agent optimized through reinforcement learning (RL), using rewards derived from LLM output quality. Across seven knowledge-intensive datasets, DynamicRAG demonstrates superior performance, achieving state-of-the-art results among models of same parameter sizes. The model, data and code are available at https://github.com/GasolSun36/DynamicRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。