用少量人工标注+LLM判断，更准估算搜索系统效果。
PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
- 结合极少量人工标注与LLM判断，重构评估框架
- 仅需100份人工标注即可降低评估误差
- 适合资源有限却需可靠评估的检索系统团队
传统搜索、排序及RAG系统质量评估依赖大量人工相关性标注。近年来，部分系统尝试使用大语言模型（LLM）作为自动化评判工具，但其内在偏差限制了直接用于指标估计。本文提出一种基于预测驱动推断（PPI）扩展的统计框架（PRECISE），通过极少人工标注与大量未标注样本结合，实现需细粒度标注的指标可靠估计。方法仅需100个经人工标注的查询和10,000个未标注样本，显著减少标注成本。针对基于LLM的查询重写应用，本框架在查询-文档层面拓展了子实例级标注的推断能力，重构指标积分空间，将计算复杂度从O(2^|C|)降至O(2^K)，其中|C|为百万级语料库规模。在多个主流检索数据集上的实验表明，该方法有效降低了关键指标Precision@K的估计方差，并在低资源场景下成功校正了LLM偏差。
原文摘要 · Abstract (English)
Evaluating the quality of search, ranking and RAG systems traditionally requires a significant number of human relevance annotations. In recent times, several deployed systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^|C|) to O(2^K), where |C| represents corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。