为生成式文档检索设计细粒度奖励机制,提升每步生成决策的准确性。
Token-Level Credit Assignment Optimization for Generative Document Retrieval

- 通过对比生成路径与标准路径的隐藏状态轨迹,实现逐标记奖励分配。
- 在多个基准上优于基线方法,提升文档检索准确率。
- 适用于需要精准生成路径优化的检索任务,如长序列文档标识符生成。
生成式检索模型通过自回归生成文档标识符(DocID)实现文档检索,这一过程本质上是序列决策问题:模型在每个解码步骤选择一个DocID标记,最终序列对应被检索文档。然而,相关性反馈仅在完整生成并映射到文档后才可获得,导致标记级生成决策与文档级监督之间存在粒度不匹配。现有强化学习方法依赖序列级奖励,对整个生成的DocID赋予相同文档级相关性信号,这种统一的信用分配掩盖了各标记决策的实际贡献,难以识别哪些决策影响检索成败。本文提出面向生成式检索的标记级信用分配(TCA),一种细粒度强化学习框架,将信用分配粒度与自回归生成对齐。不同于为整个生成的DocID分配单一奖励,TCA通过比较生成的DocID隐藏状态轨迹与来自冻结参考模型的黄金轨迹,推导出细粒度奖励。这些基于轨迹的奖励在解码步骤间提供差异化反馈,使策略能强化与目标DocID保持一致的生成路径。此外,TCA将标记级信用分配与策略优化解耦,可结合GRPO和PPO两种算法。实验表明,该方法在多个基准上持续优于基线,验证了细粒度监督在对齐DocID生成方面的有效性。
原文摘要 · Abstract (English)
Generative retrieval models perform document retrieval by autoregressively generating document identifiers (DocIDs). This process naturally forms a sequential decision problem, i.e., the model makes a sequence of token-level decisions, selecting a DocID token at each decoding step, with the resulting complete sequence identifying the retrieved document. However, relevance feedback is available only after the complete DocID has been generated and mapped to a document, resulting in a granularity mismatch between token-level generation decisions and document-level retrieval supervision. Consequently, existing reinforcement learning methods for generative retrieval rely on sequence-level rewards, assigning the same document-level relevance signal to every decoding step. Such uniform credit assignment obscures the contribution of individual token decisions, making it difficult to identify which decisions contribute to retrieval success or failure. In this paper, we propose Token-Level Credit Assignment for Generative Retrieval (TCA), a fine-grained reinforcement learning framework that aligns the granularity of credit assignment with that of autoregressive DocID generation. Unlike assigning a single reward to an entire generated DocID, TCA derives fine-grained rewards by comparing the hidden-state trajectory of each generated DocID with the gold DocID trajectory obtained from a frozen reference model. These trajectory-based rewards provide differentiated feedback across decoding steps, allowing the policy to reinforce generation paths that remain aligned with the target DocID. Moreover, TCA decouples token-level credit assignment from policy optimization and can be instantiated with both GRPO and PPO. Experiments on benchmarks show that our method consistently outperforms baselines, demonstrating the effectiveness of fine-grained supervision for aligning DocID generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。