评测10个大模型解数学题能力，发现顶尖模型比开源模型高34.5%。
Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving
- 用代码生成与执行评估模型推理，覆盖945道竞赛级数学题。
- 顶级模型gpt-4o-mini准确率达83.7%，开源模型最低仅49.2%。
- 逐词重生成可微调精度但提升效率，适合需要快速推理的场景。
大型语言模型在自然语言任务中表现优异，但在复杂数学问题求解上仍存在符号推理与输出一致性难题。本研究基于MATH数据集中的945道竞赛级题目，评估了10个参数量为7至80亿的LLM，重点考察其生成可执行Python代码的能力，涉及超过9,450次代码执行。研究采用mistral-large-2411构建评分框架，对答案进行5分制打分，以缓解数学符号不一致问题，并分析逐词重生成对结果优化的影响。结果显示，性能差距显著：最先进商业模型gpt-4o-mini得分83.7%，而表现最差的开源模型open-codestral-mamba:v0.1仅得49.2%，差距达34.5%；该差距在数论等复杂领域尤为突出。尽管逐词重生成使llama3.1:8b准确率微升0.8%，但代码执行时间缩短36.7%，体现精度与效率的权衡。所有模型在更难问题上准确率普遍下降。尽管环境受控，生成代码中安全问题不足1%，且10次尝试后仍有3.17%的问题未解决，提示混合推理策略可能更有效。
原文摘要 · Abstract (English)
Large language models (LLMs) excel in many natural language tasks, yet they struggle with complex mathemat-ical problem-solving, particularly in symbolic reasoning and maintaining consistent output. This study evalu-ates 10 LLMs with 7 to 8 billion parameters using 945 competition-level problems from the MATH dataset. The focus is on their ability to generate executable Python code as a step in their reasoning process, involving over 9,450 code executions. The research introduces an evaluation framework using mistral-large-2411 to rate answers on a 5-point scale, which helps address inconsistencies in mathematical notation. It also examines the impact of regenerating output token-by-token on refining results. The findings reveal a significant 34.5% per-formance gap between the top commercial model (gpt-4o-mini, scoring 83.7%) and the least effective open-source model (open-codestral-mamba:v0.1, scoring 49.2%). This disparity is especially noticeable in complex areas like Number Theory. While token-by-token regeneration slightly improved accuracy (+0.8%) for the model llama3.1:8b, it also reduced code execution time by 36.7%, highlighting a trade-off between efficiency and precision. The study also noted a consistent trend where harder problems correlated with lower accuracy across all models. Despite using controlled execution environments, less than 1% of the generated code was unsafe, and 3.17% of problems remained unsolved after 10 attempts, suggesting that hybrid reasoning methods may be beneficial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。