用强化学习优化检索器，让大模型回答更准确
Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG
- 将检索过程改为随机采样，建模为马尔可夫决策过程
- 引入检索历史解决多跳推理中的状态混淆问题
- 在多个数据集上显著提升RAG系统性能，适配不同规模检索器
检索增强生成（RAG）使大语言模型能够基于证据生成回答，其性能取决于检索器与大模型之间的匹配程度。检索器优化已成为替代大模型微调的高效方案。然而，现有方法存在检索优化目标与RAG整体目标不一致的问题。强化学习（RL）为解决此问题提供了可能，但将其应用于检索器优化面临两大挑战：1）确定性检索与强化学习框架不兼容；2）仅依赖查询的检索在多跳推理中引发状态混淆。为此，本文将确定性检索替换为随机采样，并将RAG建模为马尔可夫决策过程，使检索器可由强化学习优化。进一步，在每一步检索中引入检索历史作为状态，缓解状态混淆问题。在多种RAG流程、数据集及检索器规模上的大量实验表明，本方法在提升RAG性能方面具有持续有效性。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) enables large language models (LLMs) to produce evidence-based responses, and its performance hinges on the matching between the retriever and LLMs. Retriever optimization has emerged as an efficient alternative to fine-tuning LLMs. However, existing solutions suffer from objective mismatch between retriever optimization and the goal of RAG pipeline. Reinforcement learning (RL) provides a promising solution to address this limitation, yet applying RL to retriever optimization introduces two fundamental challenges: 1) the deterministic retrieval is incompatible with RL formulations, and 2) state aliasing arises from query-only retrieval in multi-hop reasoning. To address these challenges, we replace deterministic retrieval with stochastic sampling and formulate RAG as a Markov decision process, making retriever optimizable by RL. Further, we incorporate retrieval history into the state at each retrieval step to mitigate state aliasing. Extensive experiments across diverse RAG pipelines, datasets, and retriever scales demonstrate consistent improvements of our approach in RAG performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。