让大模型一次生成并排序候选结果,提升检索效率与准确性
F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking

- 将生成与排序统一在单次自回归推理中,共享一个模型主干
- 通过分阶段相对优势优化,解决生成与排序的奖励分配难题
- 在推荐和多跳问答任务上超越传统方法,无需推理时修改结构
传统检索流程分候选召回与重排两阶段,重排基于预定义候选集进行。大语言模型(LLMs)将其扩展为生成式过程:给定候选池后,可在单次自回归过程中生成子集并排序。然而,这种灵活性带来了新挑战:模型需在组合输出空间中搜索,且仅在完整排序列表生成后才能获得效用反馈。由于反馈针对完整序列,无法区分低质量结果是因未生成相关子集,还是排序错误所致,导致信用分配不明确,使端到端优化不稳定且样本效率低。现有系统常将生成与排序解耦,但这种分离与下游效用目标不一致,因重排受限于所接收的候选集。为此,我们提出统一框架,通过因子化组相对策略优化(F-GRPO)实现两者在单次自回归推演中的联合优化。该框架将策略分解为生成与排序,共享单一LLM主干,并使用与顺序无关的覆盖奖励和位置感知效用奖励联合训练。为应对由此产生的阶段特异性信用分配问题,采用两阶段序列级目标中分别计算生成与排序的组相对优势。在序列推荐与多跳问答基准测试中,F-GRPO在顶部排名性能上优于GRPO与解耦基线,超过监督型方法,且与强零样本重排器相当,推理时无需架构改动。
原文摘要 · Abstract (English)
Traditional retrieval pipelines optimize utility through stages of candidate retrieval and reranking, where ranking operates over a predefined candidate set. Large Language Models (LLMs) broaden this into a generative process: given a candidate pool, an LLM can generate a subset and order it within a single autoregressive pass. However, this flexibility introduces a new optimization challenge: the model must search a combinatorial output space while receiving utility feedback only after the full ranked list is generated. Because this feedback is defined over the completed sequence, it cannot distinguish whether a poor result arises from failing to generate a relevant subset or from failing to rank that subset correctly. This credit assignment gap makes end-to-end optimization unstable and sample-inefficient. Existing systems often address this by separating candidate generation from ranking. However, such decoupling remains misaligned with downstream utility because ranking is limited by the candidate set it receives. To bridge this gap, we propose a unified framework that performs both within a single autoregressive rollout and optimizes them end-to-end via factorized group-relative policy optimization (F-GRPO). Our framework factorizes the policy into candidate generation and ranking while sharing a single LLM backbone, and jointly trains them with an order-invariant coverage reward and a position-aware utility reward. To address the resulting phase-specific credit assignment problem, we use separate group-relative advantages for generation and ranking within a two-phase sequence-level objective. Across sequential recommendation and multi-hop question answering benchmarks, F-GRPO improves top-ranked performance over GRPO and decoupled baselines, outperforms supervised alternatives, and remains competitive with strong zero-shot rerankers, with no architectural changes at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。