用两种方法诊断大模型评分可靠性，发现评分常不一致且难易度可量化。
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
- 通过传递性分析发现评分不一致现象普遍，33%-67%文档存在矛盾循环
- 用置信集覆盖保证评分可靠性，宽度反映每篇文本的难易程度
- 不同模型评分结果一致，证明难度判断是文本特性而非随机噪声
大模型作为评分者在自然语言生成评估中日益普及，但其单个样本的可靠性仍不清晰。本文针对SummEval数据集提出双路径诊断工具：(1) 传递性分析揭示了普遍存在的输入级不一致性，尽管整体违规率较低（平均ρ = 0.8%-4.1%），但33%-67%的文档至少存在一个有向三元组循环；(2) 在1-5级李克特量表上构建分割置信集，实现理论保证的≥(1−α)覆盖率，置信集宽度作为单样本可靠性指标（rs = +0.576, N = 1,918, p < 10⁻¹⁰⁰，跨所有评委合并）。关键的是，置信集宽度在不同评委间具有稳定一致性（平均r = 0.32-0.38），表明其捕捉的是文档级难度而非评委特异性噪声。四个评委、四项标准下，两种诊断结果一致：评价维度比评委影响更大，相关性评分最可靠（平均集大小≈3.0），连贯性中等可靠（≈3.9），而流畅性和一致性仍不可靠（≈4.9）。代码、提示和缓存结果均已公开。
原文摘要 · Abstract (English)
LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: $\textbf{(1)}$ a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates ($\barρ = 0.8$-$4.1\%$), with $33$-$67\%$ of documents exhibiting at least one directed 3-cycle; and $\textbf{(2)}$ split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed $\geq(1{-}α)$ coverage, with set width serving as a per-instance reliability indicator ($r_s = {+}0.576$, $N{=}1{,}918$, $p < 10^{-100}$, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement ($\bar{r} = 0.32$-$0.38$), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size $\approx 3.0$) and coherence moderately so (avg. set size $\approx 3.9$), while fluency and consistency remain unreliable (avg. set size $\approx 4.9$). We release all code, prompts, and cached results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。