HARP为数学推理新基准，含5409道竞赛题，挑战顶级模型表现。
HARP: A challenging human-annotated math reasoning benchmark
- 构建来自美国数学竞赛的5409道人工标注数学题，分六级难度。
- 顶尖模型在最难题上准确率仅41.1%（o1-mini）和9.6%（Gemini 1.5 Pro）。
- 提供多选题与双人真解，适合研究推理能力与计算资源调度。
随着大语言模型规模扩大，数学推理成为关注焦点。然而，此前最具挑战性的评估集MATH已接近饱和（o1-mini达90.0%，Gemini 1.5 Pro达86.5%）。我们提出HARP（Human Annotated Reasoning Problems for Math），包含来自美国国家数学竞赛（A(J)HSME、AMC、AIME、USA(J)MO）的5,409道题目，其中4,780道答案可自动验证（使用SymPy等库）。题目涵盖六个难度等级，前沿模型在最难题组（197题）表现较差：o1-mini平均准确率为41.1%，Gemini 1.5 Pro仅为9.6%。数据集还包含4,110道多选题及每题平均两份人工撰写的真实解法，开辟新研究方向。我们报告了多个前沿模型的评测结果，并揭示不同模型家族在处理更难问题时会自动增加推理计算量。所有数据构建（包括爬取）与评估代码（含答案校验）均已开源，地址：https://github.com/aadityasingh/HARP。
原文摘要 · Abstract (English)
Math reasoning is becoming an ever increasing area of focus as we scale large language models. However, even the previously-toughest evals like MATH are now close to saturated by frontier models (90.0% for o1-mini and 86.5% for Gemini 1.5 Pro). We introduce HARP, Human Annotated Reasoning Problems (for Math), consisting of 5,409 problems from the US national math competitions (A(J)HSME, AMC, AIME, USA(J)MO). Of these, 4,780 have answers that are automatically check-able (with libraries such as SymPy). These problems range six difficulty levels, with frontier models performing relatively poorly on the hardest bracket of 197 problems (average accuracy 41.1% for o1-mini, and 9.6% for Gemini 1.5 Pro). Our dataset also features multiple choices (for 4,110 problems) and an average of two human-written, ground-truth solutions per problem, offering new avenues of research that we explore briefly. We report evaluations for many frontier models and share some interesting analyses, such as demonstrating that frontier models across families intrinsically scale their inference-time compute for more difficult problems. Finally, we open source all code used for dataset construction (including scraping) and all code for evaluation (including answer checking) to enable future research at: https://github.com/aadityasingh/HARP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。