对比22种重排模型，发现大模型在熟悉查询上更强，但新查询上表现不一。
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
- 对比大模型、轻量模型与零样本方法的重排效果
- 大模型在已见查询上表现更好，但新查询泛化能力参差
- 轻量模型效率高，适合资源受限场景
本文系统评估了最先进的重排方法，涵盖基于大语言模型（LLM）、轻量上下文及零样本方法，在信息检索任务中的表现。共评估22种方法，含40个变体（取决于使用的LLM），覆盖TREC DL19、DL20和BEIR等多个基准，并引入一个新数据集以测试预训练模型未见过的查询。核心目标是通过可控公平比较，判断大模型重排器与轻量模型之间是否存在性能差距，尤其在新查询上的表现差异及其成因。为分离混杂因素，分析了训练数据重叠、模型架构和计算效率的影响。结果表明，虽然大模型在已见查询上表现更优，但其对新查询的泛化能力不一致；轻量模型则展现出相当的效率优势。进一步发现，查询新颖性显著影响重排效果，暴露现有方法的局限性。
原文摘要 · Abstract (English)
In this work, we present a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods, encompassing large language model (LLM)-based, lightweight contextual, and zero-shot approaches, with respect to their performance in information retrieval tasks. We evaluate in total 22 methods, including 40 variants (depending on used LLM) across several established benchmarks, including TREC DL19, DL20, and BEIR, as well as a novel dataset designed to test queries unseen by pretrained models. Our primary goal is to determine, through controlled and fair comparisons, whether a performance disparity exists between LLM-based rerankers and their lightweight counterparts, particularly on novel queries, and to elucidate the underlying causes of any observed differences. To disentangle confounding factors, we analyze the effects of training data overlap, model architecture, and computational efficiency on reranking performance. Our findings indicate that while LLM-based rerankers demonstrate superior performance on familiar queries, their generalization ability to novel queries varies, with lightweight models offering comparable efficiency. We further identify that the novelty of queries significantly impacts reranking effectiveness, highlighting limitations in existing approaches. https://github.com/DataScienceUIBK/llm-reranking-generalization-study
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。