用规则约束大模型裁判，能显著提升推理任务准确率。
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
- 引入证据锁定的非补偿性决策规则，限制裁判随意选答案
- GSM8K准确率提升至58.2%，HotpotQA F1达25.46%
- 适合需要稳定可靠推理结果的应用场景
将大语言模型裁判部署在推理流程中，不仅评估答案质量，还决定最终输出。我们发现决策代价更多取决于嵌入的决策规则，而非裁判本身精度。在四个冻结候选池上，未经约束的深求-7B裁判仅比答案级多数投票提升1.0个百分点（GSM8K 500题），在30题确认集上反而比多数投票低10分。随后，我们将同一裁判置于证据锁定的推导-门控-修复（EL-DGR）规则下，该规则要求裁判偏好仅在有可提取证据证明时才能覆盖共识，且修复仅在无证据支持时才生效。无需更换裁判、候选或预算，EL-DGR在GSM8K达到58.2%（高于裁判56.8%、多数55.8%、首候选55.4%），在HotpotQA上达17.33 EM / 25.46 F1（高于裁判15.67/23.49）。相比首候选策略，分别提升2.8个百分点（精确麦克尼马尔检验p=0.0026）和2.00 EM（p=0.070，边缘显著）。审计显示：仅在30题中修正8次共识，且从未将正确共识转为错误答案。同时验证无效方案：七通道分解未起效，各通道缺失均不显著（所有p=1.0）。结论是：应限制裁判影响范围，而非追求其绝对准确。
原文摘要 · Abstract (English)
An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships. We show that the cost of that decision depends less on judge accuracy than on the decision rule the judge is embedded in. On frozen candidate pools from four GRPO policies, an unconstrained scalar DeepSeek-R1-7B judge buys almost nothing over answer-level majority vote (+1.0 pp on 500 GSM8K questions, +0.34 EM on 300 HotpotQA questions), and on a frozen-rule 30-question confirmation split it is 10 points worse than majority, a judge that destroys accuracy while scoring candidates confidently. We then subordinate the same judge to Evidence-Locked Derive-Gate-Repair (EL-DGR), a task-adaptive non-compensatory rule under which a judge preference may override evidence-supported consensus only with an extractive evidence certificate, and a repair only when neither alternative is certified and the repair is. With no change to the judge, the candidates, or the budget, EL-DGR reaches 58.2% on GSM8K (vs. 56.8% judge, 55.8% majority, 55.4% first candidate) and 17.33 EM / 25.46 F1 on HotpotQA (vs. 15.67/23.49, 15.33/23.19, 15.33/22.97), improving on first-candidate GRPO by +2.8 pp (exact McNemar p=0.0026) and +2.00 EM (p=0.070, borderline). A decision audit shows why: EL-DGR overturns consensus on only 8 of 30 pilot questions and never converts a correct consensus into an incorrect answer. We also report what did not work: the same seven-channel decomposition used as a step-level gated training reward is null, and corrected channel-drop ablations show no channel is individually necessary (p=1.0 throughout). The practitioner-facing finding is negative about judges and positive about admissibility, bound the judge's blast radius rather than trying to make it accurate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。