用‘以上皆非’让选择题更难，检验大模型真实推理能力
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging
- 在现有测试题中随机替换选项为‘以上皆非’，提升难度
- 模型平均得分下降12.1分，证明挑战性显著增强
- 适合评估大模型真实推理能力，尤其对需额外思考的模型
我们提出WiCkeD，一种简单方法：将多选题中的任意选项随机替换为“以上皆非”，类似教育测验常用策略。该方法可自动应用于任意现有基准测试，显著提升难度。我们在6个流行基准上应用WiCkeD，评估18个开源大语言模型。模型平均得分较原数据集下降12.1分。在3个MMLU数据集上启用思维链（chain-of-thought）时，WiCkeD版本的表现下降与直接使用模型时相当，表明其对具备增强推理能力的模型同样具挑战性。此外，WiCkeD揭示部分模型对额外推理需求更敏感，提供原基准未反映的信息。代码与数据已公开于https://github.com/ahmedselhady/wicked-benchmarks。
原文摘要 · Abstract (English)
We introduce WiCkeD, a simple method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with "None of the above", a method often used in educational tests. We show that WiCkeD can be automatically applied to any existing benchmark, making it more challenging. We apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight LLMs. The performance of the models drops 12.1 points on average with respect to the original versions of the datasets. When using chain-of-thought on 3 MMLU datasets, the performance drop for the WiCkeD variant is similar to the one observed when using the LLMs directly, showing that WiCkeD is also challenging for models with enhanced reasoning abilities. WiCkeD also uncovers that some models are more sensitive to the extra reasoning required, providing additional information with respect to the original benchmarks. We relase our code and data at https://github.com/ahmedselhady/wicked-benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。