用大模型辅助评估排名效果，显著降低误差并提升可靠性。
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
- 结合少量人工标注与大量大模型判断，通过预测驱动推理修正偏差。
- 在ESCI基准上，精度@4的估计标准误降低21%，从4.45降至3.50。
- 适合需要高可靠排名评估的生产系统，尤其适用于资源有限场景。
本文提出PRECISE，将预测驱动推理（PPI）扩展至排名评估，通过融合小规模人工标注与大规模大模型判断，生成无偏的排名指标估计。PPI在任意大模型错误分布下均保持无偏性。针对如Precision@K这类分层级指标，通过将输出空间计算量从O(2^|C|)压缩至O(2^K)，实现高效计算。在ESCI基准测试中，仅需30个手工标注结合Claude 3 Sonnet的判断，即可将Precision@4估计的标准误由4.45降至3.50（相对减少21%）。在实际生产系统中，该框架仅用100个真人标签与2小时领域专家标注，准确识别出三个系统变体中的最优方案；A/B测试验证其排名，带来日销售额提升407 bps。
原文摘要 · Abstract (English)
With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。