arXiv:2608.18719cs.AI2026-08

提出无需参考答案的裁判门诊断方法,判断大模型能否有效区分正确与错误答案。

Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

论文配图:Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization
图 1 · 摘自论文原文
  • 将裁判视为潜在求解器,用其自身能力评估答案质量。
  • 证明裁判判别力受能力与答案空间大小制约,仅当能力高于1/k时才有效。
  • 实测显示裁判准确率高不等于适用,该诊断可低成本预判闭环使用效果。

文本空间技能优化通过演化自然语言技能文档来调整冻结智能体,并通过验证门筛选候选方案。现有验证门依赖可验证奖励,限制了其在无自动验证器任务中的应用。若以大模型裁判门替代验证器,虽能突破此限制,但其是否具备有效评价信号尚不清楚。本文提出一个前置问题:在将裁判引入系统前,能否判断其评分是否能区分正确与错误答案?为此,将无参考裁判形式化为潜在求解器——其判决基于与自身推理结果的一致性,因此其评估能力受限于其求解能力。模型推导出判别力(ROC-AUC)的闭式上界,给出必要条件 $c > 1/k$,并发现边际AUC受题目难度干扰,而同题内估计量不受影响。非干预探测器记录真实优化过程中的裁判评分,未改变任何决策。实验发现:当裁判能力接近下限时判别力为随机水平,高于下限时则可用;裁判基准准确率会夸大真正关键的能力值;在闭环研究中,该诊断能预测不同类型的门控错误。结果提供一种低成本的裁判门部署前诊断工具。

原文摘要 · Abstract (English)

Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards, confining these methods to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift that restriction, but whether such a gate carries usable signal is untested. We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all? We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve. The model yields a closed-form bound on discriminability (ROC-AUC) in the judge's competence $c$ and answer-space size $k$, a necessary condition $c > 1/k$, and the result that the marginal AUC is confounded by item difficulty while a within-question estimator is not. A non-intervening probe records judge scores on genuine optimization runs without altering any decision. We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that matters; and, in a closed-loop study, that the screen predicts which kind of gating error occurs. The result is a cheap pre-deployment diagnostic for judge gates.

大模型评估技能优化裁判门无参考评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。