大模型评分器的温度设置无法保证结果稳定，可能误导安全评估结论。
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
- 未设温度和随机种子导致评分结果在相同运行中频繁波动
- 即使强制贪心解码，仍有1-2个边界案例无法复现（690次调用）
- 建议将评分分歧率作为评估工具的核心健康指标
LLM-as-judge（评分器）已成为安全评估标准组件，常用于决定下游部署。普遍认为将采样温度设为0可实现确定性评分。我们基于日本AISI开源的aisev代码库测试发现该假设在两个层面失效：第一，评估框架未设置温度或随机种子，底层服务商默认使用1.0，导致接近决策边界的项目在相同运行中反复切换通过/失败（单项目20次运行中不一致率达~50%）；第二，即使固定温度=0，仍存在非复现情况：在跨两家服务商、三种模型层级、五种采样配置的690次API调用中，有1-2个边缘案例无法稳定输出。Claude Opus 4.7/4.8已弃用温度参数，使原有缓解措施失效。研究揭示评估框架若仅报告单次结果而无方差或评分分歧度量，可能将噪声误作安全属性。我们发布了完整复现环境（690次调用，7种条件），并建议将评分分歧率作为与得分同等重要的健康指标。
原文摘要 · Abstract (English)
LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. A widespread assumption is that setting the grader's sampling temperature to 0 makes grading deterministic. We test this assumption against a real safety-evaluation codebase (Japan AISI's open-source aisev) and show it fails on two levels. First, the harness invokes its grader without setting temperature or seed; the underlying provider silently applies its default of 1.0, so items near the decision boundary flip pass/fail across identical runs (per-item disagreement up to ~50% over 20 runs). Second, pinning temperature=0 reduces but does not eliminate flips: across 690 API calls spanning two providers, three model tiers, and five sampling configurations, 1-2 of 7 borderline items remain non-reproducible even under forced greedy decoding (top_k=1). Claude Opus 4.7/4.8 has since deprecated temperature entirely, rendering the primary mitigation inapplicable to newer model generations. These findings expose a structural gap: evaluation harnesses that report single-run verdicts without variance or grader-disagreement metrics can present noise as a safety property. We release a reproduction harness (690 calls, 7 conditions) and recommend that harnesses treat grader disagreement as a first-class health metric alongside the scores themselves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。