用单元测试评估大模型数学推理，更真实考察解题能力
UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts
- 通过每题68个测试用例的单元测试框架评估数学推理
- 顶尖模型仅32.57%正确率，凸显任务难度
- 推理转代码思路提升解题质量，适合研究者与工程师参考
数学推理能力评估对推动通用人工智能发展至关重要。尽管大语言模型在解题上表现优异，但现有基准如GSM8K和MATH存在题目定义狭窄、依赖预设规则等问题，难以准确衡量模型的推理能力与泛化性。本文提出UTMath基准，一个基于大量单元测试的评估框架，涵盖9个数学领域共1,053道前沿题目，平均每题68个测试用例。该基准极具挑战性，最优模型o1-mini仅解决32.57%的问题，o1-preview为27.16%，GPT-4o为26.93%。我们还提出推理转代码思维（RCoT）方法，引导模型在生成代码前进行显式推理，从而提升解决方案的复杂度与效率。此外，我们公开了包含70,000+样本的UTMath-Train训练数据集，以支持社区进一步研究数学推理。基准可访问：https://github.com/UTMathGroup/UTMath
原文摘要 · Abstract (English)
The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and generality. This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess LLMs through extensive unit tests, with a focus on both the accuracy and generality of model responses. It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57\% of the problems, followed by o1-preview at 27.16\%, and GPT-4o at 26.93\%. Furthermore, we present the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to engage in explicit reasoning prior to code generation, thereby facilitating the production of more sophisticated solutions and enhancing overall performance and efficiency. Additionally, we also release the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning. Our benchmark can be accessed via the following link: https://github.com/UTMathGroup/UTMath
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。