对比两种纠偏LLM评分的方法，发现数据量决定选哪个。
Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
- 用贝叶斯线性模型和神经微分方程流分别建模评分校正，思路相反。
- 100组锚点下线性模型更准，1500组时流模型全面胜出。
- 数据少用线性法，数据多用流模型，可指导实际部署决策。
用大语言模型充当自动评分员虽便宜但易偏差：部分评分宽松，部分严格，中间范围压缩，冗长回答可能被高估。常见做法是事后校正：保留原评分器，在少量成对样本上拟合从原始分数到人类评分估计的转换。本文对比两种对立的建模方式：参数化的小锚点分层贝叶斯线性校正（带每分不确定性）与非参数化的神经微分方程（FFJORD）评分传输流。两者在UltraFeedback细粒度评分数据集（1700对样本，200个保留）上进行对比，校正任务分为三部分：总体均值恢复、单题准确率与分布形状匹配。核心发现是方法选择主要取决于数据预算。两者均将原始+0.71分的均值偏移降至±0.08以内（对比GPT-4参考），在100和1500个锚点下均可实现。超过1500后，两者角色互换：100锚点下，线性模型在KL散度上优一倍（0.031 vs. 0.058），MAE持平；1500锚点下，流模型在所有指标上胜出（MAE 0.320 vs. 0.359，Pearson 0.922 vs. 0.896，KL 0.026 vs. 0.037）。贝叶斯线性模型在1500前已饱和，因构造上无法拟合残余的tanh形非线性；而流模型随标注增多持续改进。研究给出生产部署的明确决策规则。
原文摘要 · Abstract (English)
[Abridged] Using a Large Language Model (LLM) as an automatic rater (LLM-as-a-judge) is cheap but potentially biased: some judges run lenient, others strict, the middle of the scale gets compressed, and verbose answers may be over-rewarded. A common remedy is post-hoc calibration: leave the cheap judge in place and, on a modest set of paired anchors, fit a transformation from raw judge scores to an estimate of the human rating. We compare two correctors that take opposing views on how this mapping should be modeled: a parametric, small-anchor hierarchical Bayesian linear correction with per-score uncertainty, and a non-parametric Neural-ODE (FFJORD) score-transport flow. Both are run head-to-head on UltraFeedback fine-grained_score (1700 paired examples, 200 held out), with calibration split into three operational sub-questions: population-mean recovery, per-item accuracy, and distributional-shape match. The headline result is that the choice between methods is primarily a data-budget question. Both correctors close the raw $+0.71$-point mean offset to within $\pm 0.08$ of the GPT-4 reference, at 100 and at 1500 anchors. Past that, the methods swap roles. With 100 anchors, the linear corrector reconstructs the human-score distribution roughly twice as well by KL divergence (0.031 vs. 0.058) and ties the flow on MAE. With 1500 anchors the flow wins on every metric (MAE 0.320 vs. 0.359, Pearson 0.922 vs. 0.896, KL 0.026 vs. 0.037). The Bayesian linear corrector saturates well below 1500 anchors: residual $\tanh$-shaped non-linearity is, by construction, structure a linear correction cannot fit. The flow keeps improving as labels grow. We translate these findings into an explicit decision rule for production deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。