大模型自评不如生成，三组数据中生成更准
Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
- 让模型自评自己生成的答案，对比生成与判断能力
- 在四组数据中，生成准确率高于自评，仅多跳任务例外
- 自评时注意力集中度仅为生成的1/5，不读候选答案
LLM-as-a-Judge与自我评估流程隐含假设：评估比生成更容易。我们在一个受控的上下文问答场景中检验这一假设，其中上下文段落是唯一信息源，每个模型对自身生成的答案进行评判，排除开放域中参数知识的干扰。在四个基准（SQuAD 2.0、DROP、HotpotQA、MuSiQue）和两个模型上测试发现，评估并非普遍更简单：在四组中的三组中，生成准确率高于自评，仅多跳任务MuSiQue例外。注意力分析显示：评估时对上下文的关注度仅为生成的3–5倍，几乎不读候选答案。LoRA微调证实该不对称性非训练偏差：生成微调导致过度接受，评估微调则损害生成能力。这些结果挑战了自我评估流程的核心假设。
原文摘要 · Abstract (English)
LLM-as-a-Judge and self-evaluation pipelines implicitly assume that evaluation is easier than generation. We test this in a controlled in-context QA setting where a context passage is the sole information source and each model judges the answer it generated, removing the parametric-knowledge confound of open-domain comparisons. Across four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) and two models, evaluation is not uniformly easier: generation accuracy exceeds self-evaluation on three of four, with multi-hop MuSiQue the exception. Attention analysis reveals why: evaluation attends to context 3--5x less than generation does and barely reads the candidate answer. LoRA fine-tuning confirms the asymmetry is not a training artifact: generation fine-tuning induces over-acceptance and evaluation fine-tuning degrades generation. These findings challenge core assumptions in self-evaluation pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。