提升大模型评分的准确性,减少系统性误差影响。
Efficient Inference for Noisy LLM-as-a-Judge Evaluation
- 提出基于高效影响函数的统一评估方法。
- 实证显示预测增强推断法方差更小,更稳定可靠。
- 适合需要高精度评估的生成模型研究者使用。
大型语言模型(LLMs)正被广泛用作生成式AI输出的自动评估工具,即“大模型作为裁判”范式。然而,大模型裁判的判断并非完美,常存在系统性非随机误差。现有两种主要方法应对此问题:(i) 基于误分类模型的直接误差校正(如Rogan-Gladen型估计器),(ii) 替代结果方法(如预测增强推断,PPI),通过在少量黄金标准人工标注上校准预测残差来纠正偏差。本文系统研究这两类方法在估计均值参数(如平均基准分数或成对胜率)时的表现。借助半参数效率理论,我们推导出基于高效影响函数(EIF)的最优估计器形式,并刻画了在何种条件下PPI类估计器的渐近方差严格小于测量误差校正方法。我们在模拟实验中验证了理论结果,并在真实数据上展示了方法效果。相关代码与比较工具已开源:https://github.com/yiqunchen/debias-llm-as-a-judge。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as automatic evaluators of generative AI outputs, a paradigm often referred to as "LLM-as-a-judge." In practice, LLM judges are imperfect predictions for the underlying truth and can exhibit systematic, non-random errors. Two main approaches have recently been proposed to address this issue: (i) direct measurementerror correction based on misclassification models such as Rogan-Gladen-style estimators, and (ii) surrogate-outcome approaches such as prediction-powered inference (PPI), which correct bias by calibrating prediction residuals on a small set of gold-standard human labels. In this paper, we systematically study the performance of these two approaches for estimating mean parameters (e.g., average benchmark scores or pairwise win rates). Leveraging tools from semiparametric efficiency theory, we unify the two classes of estimators by deriving explicit forms of efficient influence function (EIF)-based efficient estimators and characterize conditions under which PPI-style estimators attain strictly smaller asymptotic variance than measurement-error corrections. We verify our theoretical results in simulations and demonstrate the methods on real-data examples. We provide an implementation of the benchmarked methods and comparison utilities at https://github.com/yiqunchen/debias-llm-as-a-judge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。