用自回归生成文档编号,实现高效高精度排序。
Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders
- 通过逐令牌生成文档编号,提升排序表达能力。
- 理论证明自回归模型仅需常数隐层维度即可排序,优于双编码器。
- 新损失函数有效抑制无效编号生成,显著提升排名指标。
大语言模型的成功推动了生成式检索与排序的发展,旨在超越传统双编码器(DE)和交叉编码器(CE)。点对点自回归排序(ARR)让语言模型逐令牌生成文档标识符(docID),通过束搜索实现排序,兼具更强表达性与更低计算成本。然而其理论基础缺失,且标准的下一个词预测损失不适用于排序任务。本文首次证明:ARR的表达能力严格优于DE——DE需随语料规模线性增长嵌入维度才能实现任意排序,而ARR仅需常数维度即可完成。为此,我们提出SToICaL(简单令牌-项目校准损失),通过项目级重加权和前缀树边际化,基于真实相关性分配有效docID的概率质量。在WordNet和ESCI数据集上的实验表明,该损失能有效抑制无效文档编号生成,并显著提升排名指标,超越仅优化top-1的情况。
原文摘要 · Abstract (English)
The success of Large Language Models (LLMs) has motivated a shift toward generative approaches to retrieval and ranking, aiming to supersede classical Dual Encoders (DEs) and Cross Encoders (CEs). A prominent paradigm is pointwise Autoregressive Ranking (ARR), where an LLM generates document identifiers (docIDs) token-by-token to enable ranking via beam search. ARR offers the promise of superior expressivity compared to DEs while avoiding the prohibitive computational cost of CEs. However, a formal theoretical foundation for this expressive power has been missing. Moreover, the standard next-token prediction loss is rank-agnostic and inappropriate for finetuning an LLM for ranking tasks. In this paper, we first prove that the expressive capacity of ARR is strictly superior to DEs. While a DE requires an embedding dimension that grows linearly with corpus size to achieve arbitrary rankings, ARR can solve it with a constant hidden dimension. We then propose SToICaL (Simple Token-Item Calibrated Loss), a generalized rank-aware training loss for LLM finetuning. By using item-level reweighting and prefix-tree marginalization, we distribute probability mass over valid docID tokens based on their ground-truth relevance. Experiments on WordNet and ESCI datasets verify that our loss suppresses invalid docID generations and significantly improves ranking metrics beyond top-1 retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。