提出双评判协议，检测视觉语言模型在法律任务中的可解释性漏洞。
Beyond Accuracy: A Dual-Judge Evaluation Protocol for Vision-Language Models in Legally Grounded Tasks
- 用质量评分与语义等价双重标准评估模型输出
- 发现高分答案在遮挡严重时可信度骤降（最高54%-63%不匹配）
- 适合关注AI司法应用可信度的研究者和开发者
AI系统在法律问责场景中需不仅答对，还要能依法解释。现有基准和大模型评分机制虽重要，但缺乏对回答合法性的严格验证。本文提出双评判协议：一个0-10分质量评分与一个严格的二值语义等价判断，对照人工标注参考答案。以英国交通标志识别为具体任务，在7种可见度、2种遮挡模式下对4,680次评估进行测试。两评判者相关性中等（点二列相关r=0.644），揭示出不对称的Ⅱ类错误模式——在高可见度时边际错误率峰值达14.2%，但若已得高分（>7），则在重度遮挡下错误率高达54%-63%，说明高分不可信。人工核查显示，0-10分评委与普通读者判断高度一致（皮尔逊r=0.81），而等价评委更严格且单向偏严。该协议每项评估增加一次大模型调用，但能暴露单评判无法察觉的信号。代码、提示模板与完整结果均已公开。
原文摘要 · Abstract (English)
AI systems are increasingly evaluated for legally accountable settings, where correct outputs must also be justifiable against an applicable legal standard. Existing legal-AI benchmarks and LLM-as-judge protocols provide important infrastructure for measuring task performance and open-ended response quality. We contribute one additional evaluation signal: a dual-judge protocol that pairs a standard 0-10 quality judge with a strict binary semantic-equivalence judge against a human-curated reference. We study a controlled, visually grounded regulatory task - UK traffic-sign interpretation, whose meaning is a codified question with a known reference for every input - and measure not merely whether the two judges disagree (by construction they must) but how much and where. On 4,680 evaluations under seven visibility levels and two occlusion modes, the two judges are moderately associated (point-biserial r = 0.644), while revealing an asymmetric Type II pattern affecting 8.0% of all evaluations. Its distribution is instructive: the marginal rate peaks at high visibility (14.2% at v = 0.8) simply because high-scoring answers are common there, but conditioned on the answer already scoring above 7, the rate is highest under heavy occlusion (54-63% at v <= 0.3), so a high quality score is least trustworthy when the input is most degraded. We are explicit that the signal is a property of this judge and reference: a 49-row human check shows the 0-10 judge aligns closely with everyday-reader judgement (Pearson r = 0.81; r = 0.80 with the LLM accuracy sub-score), while the equivalence judge is fairly but one-directionally stricter. The protocol adds one LLM call per evaluation and surfaces a signal single-judge protocols do not report. We release the prompt template, occluded variants, and full evaluation results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。