改进多选题评估方法，避免因答案表述不同导致的误判。
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
- 用多种改写版本测试答案，减少对特定表述的依赖。
- 相同知识模型间虚假性能差距从2分降至1分以下。
- 适合评估大模型真实能力，尤其对高阶模型更可靠。
多选题评测是评估预训练大模型的标准方法，但其依赖似然分数会因答案的精确表述（表面形式）而产生偏差，混淆模型对特定短语的熟悉程度与真实能力。我们通过一个10亿至80亿参数模型的受控实验验证此问题：尽管这些模型具备完全相同的知识，标准评分仍错误地报告超过2分的性能差距。为此，我们提出ParaEval评估框架，对每个选项使用多个改写版本进行提问。通过采用模型在所有改写中表现最佳的结果作为评分，该方法将虚假性能差距压缩至1分以下。我们进一步验证，这一评估误差及ParaEval的改进效果在前沿的700亿和1200亿参数开源模型中依然存在。最终，ParaEval提供了一种稳健且高效的评估方式，真正反映模型的底层能力而非表面形式的熟悉度。
原文摘要 · Abstract (English)
Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically, standard scores are highly sensitive to the exact phrasing (surface form) of the answers, conflating a model's familiarity with a specific phrase with its actual capability. We demonstrate this flaw using a controlled testbed of 1B-8B models trained on the same knowledge. Despite having identical knowledge, standard metrics falsely report a performance gap of over 2 points. To solve this, we propose ParaEval, an evaluation framework that queries models using multiple paraphrases per answer option. By scoring each model based on its most favorable phrasing, ParaEval successfully reduces the false performance gap to below 1 point. We confirm that these evaluation artifacts, and the improvements from ParaEval, persist in frontier 70B and 120B open-source models. Ultimately, ParaEval provides a robust and efficient way to evaluate true underlying capability rather than surface-form familiarity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。