新基准测试大模型复杂推理与捷径学习，区分顶尖模型差异。
MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
- 引入多正确答案题目，检验模型复杂推理能力。
- 六款顶级模型表现差距明显，揭示推理与偏见差异。
- 新增捷径选择率等指标，深入分析模型行为机制。
现有大模型评估基准难以区分性能顶尖的模型，亟需更具挑战性的评测框架。本文提出MMLU-Pro+，在MMLU-Pro基础上增强，用于评估大模型的捷径学习与高阶推理能力。通过引入跨领域、存在多个正确答案的问题，MMLU-Pro+考验模型在复杂情境下的推理能力，并抑制简单化解题策略。结果表明，该基准在保持原难度的同时，显著提升了对模型的区分力，尤其在多正确答案场景中效果更优。我们提出快捷方式选择比率和正确答案对识别率等新指标，深入揭示模型行为与锚定偏见。对六款先进大模型的评估显示显著性能差异，凸显其推理能力与偏见敏感度的不同。数据集与评估代码已开源至：https://github.com/asgsaeid/mmlu-pro-plus。
原文摘要 · Abstract (English)
Existing benchmarks for large language models (LLMs) increasingly struggle to differentiate between top-performing models, underscoring the need for more challenging evaluation frameworks. We introduce MMLU-Pro+, an enhanced benchmark building upon MMLU-Pro to assess shortcut learning and higher-order reasoning in LLMs. By incorporating questions with multiple correct answers across diverse domains, MMLU-Pro+ tests LLMs' ability to engage in complex reasoning and resist simplistic problem-solving strategies. Our results show that MMLU-Pro+ maintains MMLU-Pro's difficulty while providing a more rigorous test of model discrimination, particularly in multi-correct answer scenarios. We introduce novel metrics like shortcut selection ratio and correct pair identification ratio, offering deeper insights into model behavior and anchoring bias. Evaluations of six state-of-the-art LLMs reveal significant performance gaps, highlighting variations in reasoning abilities and bias susceptibility. We release the dataset and evaluation codes at \url{https://github.com/asgsaeid/mmlu-pro-plus}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。