融合不确定性工具在法律预测中不如直接用大模型，关键价值是可信决策而非更准预测。
Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion
- 将大模型与贝叶斯、戴明-沙弗等融合，反而加剧校准误差
- 融合后对歧视案件的预测性能未提升，原始模型已最优
- 真正价值在于用置信度筛选可自动化案件，降低错误率
本文在1000个真实欧洲人权法院案例（来自LexGLUE和FairLex）上测试了多种不确定性融合方法（证据图+信念传播、贝叶斯序贯更新、戴明-沙弗组合、置信区间预测）在案件结果预测中的效果。对比了两个前沿大模型（Claude Opus 4.8 和 GPT-5.5）作为逐事实证据估计器的三种方案：(A) 原始大模型，(B) 经融合管道处理的大模型，(C) 词频基线通过相同管道。共进行约4750次测试发现：(1) 在歧视类案件中，融合管道未提升性能（AUROC约0.83），直接使用大模型是最强判别器；(2) 朴素组合大模型与贝叶斯、戴明-沙弗融合使校准误差翻倍（ECE从约0.16升至0.46），源于先验不匹配；(3) 戴明-沙弗在长链推理中极度不安全，低准确率下仍高自信地输出错误标签，建议移除；(4) 管道真正价值在操作层面：经剔除戴明-沙弗、重新校准并应用类别条件风险控制后，系统实现96.8%自动通过率，仅0.5%错误漏过，96.3%被标记审查，远优于未调参基线（85.9%/3.8%/72.1%）。法律领域中，融合管道的价值在于校准信任，而非更锐利的预测。
原文摘要 · Abstract (English)
A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline. We test this on 1,000 real European Court of Human Rights cases from LexGLUE and FairLex, predicting whether the Court found a Convention violation from the case's fact paragraphs. We compare three families across two frontier LLMs (Claude Opus 4.8 and GPT-5.5) as per-fact evidence estimators: (A) the raw LLM, (B) the LLM routed through the fusion pipeline, and (C) a term-frequency baseline through the same pipeline. Across roughly 4,750 tests we find: (1) on discrimination (AUROC around 0.83) the pipeline yields no improvement over either the raw LLM or the baseline; a frontier LLM used directly is the strongest single discriminator. (2) Naively composing an LLM with Bayesian-odds and Dempster-Shafer fusion more than doubles calibration error (ECE from about 0.16 to 0.46) via a prior-mismatch mechanism that replicates across both models. (3) Dempster-Shafer fusion is actively unsafe on long chains, committing confidently to wrong labels at below-chance accuracy; we recommend removing it. (4) The pipeline's genuine value is operational: routed through a conformal selective-prediction layer, the system decides which cases to automate and which to escalate. After removing Dempster-Shafer, recalibrating, and applying class-conditional risk control on the full 1,000-case set, the tuned engine auto-clears at 96.8 percent accuracy with 0.5 percent errors escaping and 96.3 percent caught for review, versus 85.9 / 3.8 / 72.1 for an untuned baseline. The contribution of such pipelines in law is calibrated trust, not sharper prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。