用最新论文自动构建可演化的数学推理评测集
EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
- 从最新发表的数学论文中自动生成可执行的推理题
- 实测顶尖大模型在前沿数学任务上仍有巨大差距
- 适合关注数学推理与动态评测的研究者
当前大语言模型数学推理评估多依赖静态基准,或来自竞赛题,或需昂贵的人工标注,导致研究级数学覆盖不足且性能迅速饱和。本文提出一种全自动、基于定理的评估流水线,可直接将近期同行评审的数学文献转化为可执行、可验证的推理任务。该流程识别构造性或量化结果,将其转化为参数化问题模板,并通过执行验证生成确定性解,实现无需大规模人工参与的可扩展、可复现、持续更新的评估。该方法支持时间延展性、内在正确性检验及子领域定制。基于此构建的动态评测集名为EternalMath。实验显示,现有顶尖大模型在前沿数学任务上仍存在显著差距,表明数学推理能力尚未饱和,亟需与人类发现同步演进的评估方法。
原文摘要 · Abstract (English)
Current evaluations of mathematical reasoning in large language models (LLMs) are dominated by static benchmarks, either derived from competition-style problems or curated through costly expert effort, resulting in limited coverage of research-level mathematics and rapid performance saturation. We propose a fully automated, theorem-grounded pipeline for evaluating frontier mathematical reasoning, which directly transforms recent peer-reviewed mathematical literature into executable and verifiable reasoning tasks. The pipeline identifies constructive or quantitative results, instantiates them into parameterized problem templates, and generates deterministic solutions through execution-based verification, enabling scalable, reproducible, and continuously updatable evaluation without reliance on large-scale expert authoring. By design, this approach supports temporal extensibility, intrinsic correctness checking, and domain-specific customization across mathematical subfields. Applying this pipeline yields \textbf{EternalMath}, an evolving evaluation suite derived from contemporary research papers. Experiments with state-of-the-art LLMs reveal substantial performance gaps, indicating that mathematical reasoning at the research frontier remains far from saturated and underscoring the need for evaluation methodologies that evolve in step with human mathematical discovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。