构建可演进的数学猜想基准,助力AI发现新定理
Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
- 将2615个数学问题形式化为Lean 4语言,含1029个开放猜想
- 已用该基准促成新数学发现,包括解决多个未解猜想
- 支持人机协作验证,适合研究自动化推理与数学AI者
随着自动化推理系统快速发展,亟需高质量的形式化数学问题来评估其能力。为此,我们提出Formal Conjectures,一个在Lean 4中形式化的动态基准库,包含2615个数学问题,涵盖1029个当前开放的研究猜想,提供零污染的数学证明发现测试集;另有836个已解决的问题用于证明自动形式化训练。该库建立结构化接口,连接数学家、AI系统与求解者,实现协同验证。实际应用中,该基准已助力发现新成果,包括解决若干长期未解猜想。通过开源协作与AI生成的证明/反证作为审计机制,持续提升形式化质量。我们还提供了标准化评估方案,并报告了冻结子集上的基线结果,展现可攀登的性能信号,反映当前自动化推理在研究级数学中的前沿水平。
原文摘要 · Abstract (English)
As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their capabilities. To address this, we present Formal Conjectures, an evolving benchmark of currently 2615 mathematical problem statements formalized in Lean 4. Sourced from areas of active mathematical research, the dataset features 1029 open research conjectures providing a zero-contamination benchmark for mathematical proof discovery, and 836 solved problems for proof autoformalization. Notably, the repository provides a structured interface connecting mathematicians who formalize and clarify problems with the AI systems and humans attempting to solve them. Demonstrating its immediate utility, the benchmark has already been leveraged to make new mathematical discoveries, including the resolution of open research conjectures. We describe our approach to ensuring the correctness of these formalizations in a collaborative open-source project where contributions stem from an active community. In this framework, AI-generated proofs and disproofs serve as a valuable auditing mechanism to iteratively improve the fidelity of the benchmark. Finally, we provide a standardized evaluation setup and report baseline results on frozen evaluation subsets, demonstrating a climbable signal that measures the current frontier of automated reasoning on research-level mathematics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。