用鲁棒统计方法改进大模型评分集体,对抗有偏评委提升评估准确性。
RoPoLL: Robust Panel of LLM Judges

- 以几何中位数替代平均值,实现无需调参的鲁棒评分聚合。
- 在50%恶意干扰下仍保持稳定,跨维度攻击中性能领先19%。
- 小规模团队(3个38B模型)胜过675B大模型,适合资源有限场景。
LLM评委会(PoLL)作为多模型共识评分机制已广泛应用,但其统计特性尚不明确。本文在Huber污染模型下证明:只要一个评委出现典型偏差(如模式坍缩、迎合、安全拒绝),无论评委会规模多大,都会导致无界偏差。为此提出RoPoLL,保留原评委会结构,但将聚合方式替换为几何中位数(GM),实现零调参、最优有限样本断裂点1/2。理论分析表明,其误差率与最小极大下界在σ√(d/N)量级一致,仅在断裂底限上差√d倍——这是多项式时间算法与不可行的Tukey半空间中位数间的统计-计算差距。在13个开源模型(4B-675B)、3个奖励模型基准及4种污染场景(最高达50%)测试中,RoPoLL在所有有偏污染类型下均优于PoLL:跨维攻击下性能提升约19%,对重尾拜占庭攻击提升数个数量级。3个38B模型组成的委员会,在30%双峰随机污染下,于HelpSteer-2上超越Mistral-Large-3(675B)1.31倍,参数量优势达18倍,且通过噪声真实标签控制验证该优势源于对抗性偏差而非普通误差。
原文摘要 · Abstract (English)
The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood. We formalize the LLM Jury under the Huber contamination model and show that PoLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal). Framing jury consensus as classical robust mean estimation, we propose RoPoLL (Robust Panel of LLM-as-Judge), which preserves the PoLL panel but replaces the aggregation function with a robust mean estimator, instantiated with the geometric median (GM): tuning-free, with the optimal finite-sample breakdown point 1/2. A finite-sample error bound and a matching information-theoretic minimax lower bound agree on the parametric rate sigma*sqrt(d/N) and differ on the breakdown floor by a factor of sqrt(d), a statistical-computational gap that polynomial-time RoPoLL pays relative to the intractable Tukey halfspace median. Across 13 open-weight judges (4B-675B), three reward-model benchmarks, and four corruption regimes at rates up to 50%, RoPoLL dominates PoLL on every biased corruption type: by about 19% on cross-dimensional attacks at matched compute, and by orders of magnitude on heavy-tailed Byzantine adversaries. A 3-judge RoPoLL committee at 38B beats Mistral-Large-3 (675B) by 1.31x on HelpSteer-2 under 30% bimodal-random corruption, an 18x parameter advantage at better accuracy; a Noisy-GT control confirms the premium is paid against biased contamination, not benign imprecision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。