提出一套评估大模型判官稳定性的压力测试框架。
Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence

- 设计三维度压力测试:重述稳定性、单轮抗辩力、多轮持续抗压性。
- 9个前沿模型在压力下判罚翻转率达25%~91%,多数改判违背真实答案。
- 发现集体意见强度可预判判官易被操纵程度,适合评测与安全研究者参考。
大模型判官已成为模型评估、在线评分和奖励建模的核心基础设施。当前判官通常通过黄金数据集上的准确率进行验证,但准确率无法反映其在重新提示、挑战或持续施压下的稳定性。本文提出「Wiggle框架」,一种统一的命题稳定性压力测试方法,从三个维度衡量判官鲁棒性:机械一致性(对重述与重构的稳定性)、单轮确信度(对单一挑战的稳定性)以及多轮持续性(对持续或自适应压力的抗性)。我们使用该框架在14项任务上评估9个前沿模型,涵盖安全、毒性、AI写作检测及政治回应评价。所有模型作为判官均表现出显著波动——静态施压下判罚翻转率为25%~71%,面对对抗性大模型说服时高达62%~91%。关键发现是,成功改变判官结论的压力几乎总是违背真实答案的净腐蚀性行为。此外,我们识别出基线评委多数意见强度是预测判官易动摇性的最强单因素信号。本研究首次实现跨数据集的判官机械性、顺从性与可说服性测试的直接比较。
原文摘要 · Abstract (English)
LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling. Judges are typically validated by accuracy on golden data, but accuracy says little about whether they are stable under re-prompting, challenge, or sustained pushback. We introduce the \emph{Wiggle Framework}, a unified stress test for epistemic stability in LLM judges. The framework decomposes judge robustness along three dimensions: Mechanical Consistency (stability under re-prompting and reframing), Single-turn Conviction (stability under a single challenge), and Multi-turn Persistence (stability under sustained or adaptive pressure). We use the framework to study 9 frontier models across 14 judging tasks spanning safety, toxicity, AI writing detection, and political-response evaluation. Every model exhibits substantial wiggle as a judge --- flipping verdicts 25--71\% of the time under static pushback, and 62--91\% with an adversarial LLM persuader. Critically, we find that pressure that succeeds in changing a judge's verdict is almost always net-corrupting with respect to ground truth. Beyond the framework itself, we identify baseline jury majority strength as the most effective single-shot signal for anticipating which items wiggle. Taken together, this is the first apples-to-apples cross-dataset comparison of mechanical, conformity, and persuadability tests in a judging context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。