用教育评估标准检测多选题数据集的三大缺陷，提升评测可信度。
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
- 基于教育评分规则，用大模型自动识别多选题的污染、提示线索和语法错误
- 发现47%的TruthfulQA题曾在线出现，HellaSwag全题违反写作规范
- 揭示错误会扭曲模型性能评估，且修复旧问题可能引入新缺陷
多选题问答（MCQA）是自然语言处理的常用评测方式，但现有基准缺乏严格质量控制。本文提出BenchMarker，一个受教育评估启发的工具包，利用大模型判官识别三类常见多选题缺陷：1）污染：题目与网络内容完全一致；2）捷径：选项中存在可被猜测的提示线索；3）书写错误：依据19条教育评分标准判断的结构或语法问题。通过人工标注验证后，我们对12个基准进行审计，发现：1）缺陷在自动生成和众包数据中尤为普遍——TruthfulQA有47%题目已在线出现，HellaSwag全部违反多项写作规则；2）污染题常虚高准确率，书写错误则降低分数并改变排名，影响超过随机水平；3）以往修复措施虽解决特定问题（如用大模型生成干扰项降低准确率），却意外引入新缺陷（如不合理干扰项、多个正确答案）。总体而言，多选题缺陷严重损害了NLP评测的可靠性，而教育研究提供了改进路径。我们开源BenchMarker，推动两领域融合，优化多选题基准设计。
原文摘要 · Abstract (English)
Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges to flag three common MCQ flaws: 1) contamination: items appearing exactly online; 2) shortcuts: cues in the choices that enable guessing; and 3) writing errors: structural/grammatical issues based on a 19-rule education rubric. We validate BenchMarker with human annotations, then run the tool to audit 12 benchmarks, revealing: 1) flaws persist in MCQA benchmarks, especially automatically-made and crowdsourced data - we detect 47% of TruthfulQA appears online and 100% of HellaSwag violates multiple writing rules; 2) contaminated MCQs tend to inflate accuracy, while writing errors tend to lower it and change rankings beyond random; and 3) prior benchmark repairs address their targeted issues (i.e., lowering accuracy with LLM-written distractors), but inadvertently add new flaws (i.e. implausible distractors, many correct answers). Overall, flaws in MCQs degrade NLP evaluation, but education research offers a path forward. We release BenchMarker to bridge the fields and improve MCQA benchmark design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。