提出新评估标准，精准衡量大模型重排器的效率与效果。
Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers
- 用每拍（PetaFLOP）的排序质量与查询处理数评估效率。
- 实测多种模型，揭示架构对性能-效率权衡的影响。
- 适合关注部署效率的工程研究者与模型优化团队。
大语言模型（LLM）在信息检索重排任务中表现优异，但其高计算需求限制了实际应用。现有评估方法依赖延迟、前向传播次数、输入输出标记数等代理指标，这些指标受硬件和运行参数影响，且未考虑模型规模，难以准确反映效率-效果权衡。为此，本文提出针对 LLM 重排器的新评估框架：RPP（每 PetaFLOP 的排序指标得分，如 NDCG/MRR），衡量单位算力下的排序质量；QPP（每 PetaFLOP 处理的查询数），衡量单位算力下的吞吐量。同时开发可解释的 FLOPs 估算器，无需实验即可预估算力消耗。基于此框架，系统评测了多种架构的 LLM 重排器，揭示效率-效果关系，推动该问题在社区中的重视。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently been applied to reranking tasks in information retrieval, achieving strong performance. However, their high computational demands often hinder practical deployment. Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency, the number of forward passes, input tokens, and output tokens. However, these metrics depend on hardware and running-time choices (\eg parallel or not, batch size, etc), and often fail to account for model size, making it difficult to interpret and obscuring the evaluation of the efficiency-effectiveness tradeoff. To address this issue, we propose \ours\footnote{https://github.com/zhiyuanpeng/EER-FLOPs.} for LLM-based rerankers: RPP (ranking metrics per PetaFLOP), measuring how much ranking quality (e.g., NDCG or MRR) a method achieves per PetaFLOP, and QPP (queries per PetaFLOP), measuring how many queries can be processed per PetaFLOP. Accompanied by the new metrics, an interpretable FLOPs estimator is developed to estimate the FLOPs of an LLM-based reranker even without running any experiments. Based on the proposed metrics, we conduct comprehensive experiments to evaluate a wide range of LLM-based rerankers with different architectures, studying the efficiency-effectiveness trade-off and bringing this issue to the attention of the research community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。