用首个生成词的逻辑值实现快速列表重排序,大幅降低计算开销。
An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking
- 仅用首个词的输出做重排序,极大减少推理时间。
- 在TREC DL19-22数据集上表现优于传统方法,跨领域效果稳定。
- 适合作为需要高速重排序的系统集成,如搜索与推荐场景。
近期研究显示大型语言模型(LLMs)在列表重排序任务中表现出色,但其高计算成本仍是广泛采用的障碍。此外,传统的语言建模目标并不适合重排序任务。FIRST是一种新方法,通过引入学习排序目标并仅利用首个生成词的逻辑值,显著降低了传统LLM重排序器的推理延迟。本研究将FIRST扩展至TREC深度学习数据集(DL19-22),验证其在多种领域的鲁棒性。我们考察了不同第一阶段检索器对FIRST重排序器的影响,观察到收益递减现象,且规律与传统LLM重排序器一致。通过将FIRST目标应用于更广泛的骨干模型,我们实现了超越原始实现的效果。实验表明,仅使用单个词逻辑值进行快速重排序不会损害跨域重排序质量。为更准确量化原研究中的计算节省,我们测量并比较延迟,发现各类模型和基准下延迟降低21%-42%。此外,尽管语言建模训练隐式提升了零样本单词重排序性能,但我们的实验也提出了疑问:语言建模预训练是否可能阻碍后续以FIRST目标进行的微调。这些发现为未来更高效、有效的列表重排序应用铺平道路。
原文摘要 · Abstract (English)
Recent advances have demonstrated that large language models (LLMs) excel as listwise rerankers, but their high computational demands remain a barrier to widespread adoption. Further, the traditional language modeling (LM) objective is not ideally suited for reranking tasks. FIRST is a novel approach that addresses these challenges by integrating a learning-to-rank objective and leveraging the logits of only the first generated token, thereby significantly reducing inference latency compared to traditional LLM rerankers. In this study, we extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. We investigate the influence of different first-stage retrievers on FIRST rerankers, observing diminishing returns and patterns consistent with traditional LLM rerankers. Through applying the FIRST objective to a broader range of backbone models, we achieve effectiveness surpassing the original implementation. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality. To better quantify the computational savings in the original study, we measure and compare latency to find a 21%-42% gain across various models and benchmarks. Moreover, while LM training implicitly improves zero-shot single-token reranking, our experiments also raise questions about whether LM pre-training may hinder subsequent fine-tuning with the FIRST objective. These findings pave the way for more efficient and effective listwise reranking in future applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。