大模型能判断答案对错，却难写出完整正确的参考答案集。
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
- 用单次生成不推理的模式让模型写标准答案，效果差
- 模型判断正确率高（F1 0.74-0.90），但自编答案仅通过19%-42%真解
- 主要问题在遗漏而非错误，难以被审查发现
语言模型正从答题者转向评判者，负责编写测试集、答案键和评分规则。我们评估其在真实部署协议下的能力，发现其表现不足：在四个基准上——两个有限真值构造、一个强化可执行参考（HumanEval+/MBPP+）、一个显式不完整的词汇参考（WordNet）——模型判断候选答案的能力远超其自动生成完整答案集的能力。在算法性构造中，差距达F1 +0.34至+0.29，且不随参数量增加而缩小；在代码任务中，模型判别准确率已达F1 0.74–0.90，但生成的测试集仅允许19%–42%的真正确解通过。对照实验表明，若让模型输出判定谓词而非完整集合，其性能可达F1约0.99。失败并非因知识缺失或表达困难，而是无法实现规范所诱导的完整解集。主要错误为遗漏，难以审计：过包容易被质疑，遗漏则需作者自行发现。模型检测到人为加入的过包容的概率是遗漏的6–7倍，实际部署中43,227项样本的遗漏问题比例高达10:1。在RLVR框架中，自动生成的评分键相比精确真值损失1.9分，与WordNet相比损失18.5分（六组种子，p=0.031）。通过已知正确样本校验过滤假拒率，可将误拒从58%–92%降至最多5%，但仅保留5%–39%的原测试集。通过修正错误预期值为参考执行结果，测试集产出效率提升3.3–10.6倍。
原文摘要 · Abstract (English)
Language models are increasingly promoted from examinees to examiners: they write the test suites, answer keys, rubrics, and reward functions that define correctness for other systems. We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with, one-shot greedy authoring with no test-time reasoning. Across four reference constructions - two with complete finite truth, one with a hardened executable reference (HumanEval+/MBPP+), one with an explicitly incomplete lexical reference (WordNet) - models judge whether a candidate belongs far better than they author the set itself. On the incompleteness-proof algorithmic construction the gap is +0.34 to +0.29 F1 over a 24x parameter range and does not close; on executable code, models judging at F1 0.74-0.90 author suites admitting only 19-42% of oracle-correct solutions. A control locates the deficit: asked to emit the predicate rather than its extension, the same models reach F1 about 0.99. The failure is not missing knowledge or an inability to specify, but an inability to materialise the region a specification induces. The dominant error is omission, which resists audit: an over-inclusion is a token a reviewer can challenge, a missing member an absence whose discovery is the authoring problem itself. Models detect planted over-inclusions 6-7x more often than planted omissions, and a production deployment of 43,227 items fails omission-first at 10:1. Wired into RLVR, an authored key costs 1.9 points of accuracy against an exact oracle and 18.5 WordNet-relative (six paired seeds, p=0.031). Gating authored verifiers on a known-correct probe cuts false rejection from 58-92% to at most 5%, but keeps only 5-39% of suites. Repairing them instead, by rewriting each wrong expected value to what a reference execution returns, raises yield 3.3-10.6x across four author families.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。