用大模型反馈优化重排器，让检索结果更符合生成需求。
Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
- 通过强化学习让重排器直接学习大模型的生成反馈
- 在多个知识密集型任务上超越现有最强基线模型
- 无需人工标注，适配不同大模型和查询扩展模块
重排器在检索增强生成中至关重要，但现有模型通常仅基于静态人工标注的相关性标签独立训练，与下游生成过程脱节。这种分离导致信息检索指标识别的相关文档，往往无法满足大模型生成精准答案的实际需求。为此，我们提出重排偏好优化（RRPO），一种将重排与大模型生成质量直接对齐的强化学习框架。通过将重排建模为序列决策过程，利用大模型反馈优化上下文实用性，从而无需昂贵的人工标注。为保障训练稳定性，引入参考锚定确定性基线。在知识密集型基准上的大量实验表明，RRPO显著优于强基线，包括强大的列表级重排器RankZephyr。进一步分析显示该框架具备高度泛化能力：可无缝适配不同大模型（如GPT-4o），可与Query2Doc等查询扩展模块正交集成，并在使用噪声监督信号时仍保持鲁棒性。
原文摘要 · Abstract (English)
Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation. However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process. This isolation leads to a fundamental misalignment: documents identified as topically relevant by information retrieval metrics often fail to provide the actual utility required by the LLM for precise answer generation. To bridge this gap, we introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality. By formulating reranking as a sequential decision-making process, RRPO optimizes for context utility using LLM feedback, thereby eliminating the need for expensive human annotations. To ensure training stability, we further introduce a reference-anchored deterministic baseline. Extensive experiments on knowledge-intensive benchmarks demonstrate that RRPO significantly outperforms strong baselines, including the powerful list-wise reranker RankZephyr. Further analysis highlights the versatility of our framework: it generalizes seamlessly to diverse readers (e.g., GPT-4o), integrates orthogonally with query expansion modules like Query2Doc, and remains robust even when trained with noisy supervisors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。