让大模型自己评估它生成的搜索结果质量，提升预测准确性。
Can LLM Rerankers Predict Their Own Ranking Performance?
- 利用大模型自身生成的排序一致性与自评信心进行性能预测
- 自洽性指标表现媲美顶尖方法且更校准，直接自评信心严重高估
- 提出轻量级新方法，仅增少量输出即可获得可靠质量估计
检索效果在不同查询间差异显著，因此在获得相关性判断前预估排序质量至关重要。查询性能预测（QPP）旨在解决此问题，但现有方法多依赖检索或重排序后的外部预测器。本文研究重排序器内部的QPP：大模型能否评估其刚生成的排序质量？我们探索了免训练与基于训练的方法。免训练下，考察采样排序间的指标自洽性及模型直接生成的口头置信度。在TREC Deep Learning 2019–2022数据集上，使用四种LLM的实验表明，自洽性表现优于或媲美当前最优方法，且几乎所有场景下更校准；而直接口头置信度严重过自信。为改进该问题，我们提出两种监督方法：Verb-Num与Verb-List，使大模型重排序器仅需增加少量输出标记即可生成校准良好的排序质量估计。
原文摘要 · Abstract (English)
Retrieval effectiveness varies substantially across queries, making it important to estimate ranking quality before relevance judgments are available. Query performance prediction (QPP) addresses this need, but most existing methods rely on external predictors after retrieval or reranking. In this paper, we study \textit{reranker-internal QPP}: can an LLM reranker estimate the quality of the ranking it has just produced? We investigate both training-free and training-based approaches. For training-free estimation, we examine metric-specific self-consistency across sampled rankings and verbalized confidence produced directly by the reranker. Experiments on TREC Deep Learning 2019--2022 with four LLMs show that self-consistency is competitive with the state-of-the-art (SOTA) approach and better calibrated in almost all settings, while direct verbalized confidence is severely overconfident. To improve verbalized confidence, we propose two supervised methods, Verb-Num and Verb-List, which enable LLM rerankers to produce calibrated ranking-quality estimates with only a few additional output tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。