用先验排序减少无效比较，让大模型零样本重排序更快更准。
Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion
- 基于初始排序结果，只关注可能提升的候选文档，减少冗余计算。
- 在多个模型上实现查询时间降31%、推理次数减23%，效果略提升。
- 适合追求高效零样本重排序的系统开发者或研究者。
本研究对Zhuang等人提出的集合提示方法在大语言模型零样本排序中的应用进行了全面复现与拓展分析。对比传统的点式、成对和列表式方法，实验验证了该方法在计算效率与排序效果间的权衡。在此基础上，提出新方法Setwise Insertion，利用初始文档排序作为先验知识，聚焦于更可能改进结果的候选文档，从而减少不必要的比较和不确定性。在Flan-T5、Vicuna和Llama等多模型上的实验表明，Setwise Insertion相较原方法实现31%的查询时间降低、23%的模型推理次数减少，且重排序效果略有提升。结果表明，将先验排序信息融入集合提示，可显著提升零样本文档重排序的效率与准确性。
原文摘要 · Abstract (English)
This study presents a comprehensive reproducibility and extension analysis of the Setwise prompting methodology for zero-shot ranking with Large Language Models (LLMs), as proposed by Zhuang et al. We evaluate its effectiveness and efficiency compared to traditional Pointwise, Pairwise, and Listwise approaches in document ranking tasks. Our reproduction confirms the findings of Zhuang et al., highlighting the trade-offs between computational efficiency and ranking effectiveness in Setwise methods. Building on these insights, we introduce Setwise Insertion, a novel approach that leverages the initial document ranking as prior knowledge, reducing unnecessary comparisons and uncertainty by focusing on candidates more likely to improve the ranking results. Experimental results across multiple LLM architectures (Flan-T5, Vicuna, and Llama) show that Setwise Insertion yields a 31% reduction in query time, a 23% reduction in model inferences, and a slight improvement in reranking effectiveness compared to the original Setwise method. These findings highlight the practical advantage of incorporating prior ranking knowledge into Setwise prompting for efficient and accurate zero-shot document reranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。