证据锁定会降低大模型评分一致性，反而影响判断可靠性。
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
- 将证据固定在多轮对话中，替代原始答案参与判断
- 证据锁定使评分与人类偏好一致率下降4-6个百分点
- 适合关注可审计性的场景，但不适合核心评估流程
大模型作为评判者常被要求先提取标准和证据，再做出选择。这一流程假设中间记录能保留后续判断所需信息。对于具备推理能力的模型，可见字段顺序不能反映内部决策顺序，因此我们测试了一种可观测替代方案：在一次调用中保留证据，并将其作为下一次调用的唯一输入。在HelpSteer3、FeedbackQA和CoVal三个数据集上，基于Claude Sonnet 4.5和GPT-5，我们对比了标准成对评判、结构化单次评判、双次证据锁定以及三次逐点锁定。结果表明，证据锁定使评分与已发布人类偏好的一致性下降4至6个百分点，答案顺序不一致性上升8至10个百分点，相较于结构化单次评判。逐点锁定同样有害，而结构化证据提取仍接近标准评判表现。该结论在两种模型及所有数据集上均成立。持久化的证据虽有助于审计，但不应在决策时替代原始答案。
原文摘要 · Abstract (English)
LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。