构建细粒度数学定理证明评估基准,揭示大模型在复杂推理中的真实表现。
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics

- 基于Lean4设计双版本基准,包含主定理与自动生成的支撑子定理
- 模型在带前提的测试中性能显著提升,但仍依赖冗长低效的证明路径
- 引入覆盖度与令牌效率指标,精准捕捉证明过程中的进展与偏差
大型语言模型在形式化证明基准上已取得优异成绩,但现有评估多集中于竞赛类问题,难以反映模型在更长、依赖关系更复杂的数学推导中的表现。本文提出TheoremBench,一个基于Lean4的新型基准,涵盖近一百个经典定理。该基准分为两种互补形式:基础版每例仅含一个目标定理;预设版则将每个定理扩展为包含主定理及其自动提取的支撑子定理组成的结构化任务族。此设计不仅可评估模型是否从零开始证明最终定理,还可衡量其在定理内部证明结构中的部分进展。实验表明,显式前提显著提升了具备Lean4能力的模型表现。为实现全面评估,本文引入定理级覆盖度与令牌效率指标,揭示了当前证明器对简单子定理的强偏好,以及通过冗长战术序列而非紧凑证明计划解题的普遍现象。TheoremBench为形式化推理能力提供了更精细的评估视角,并强调了结构化基准设计的重要性。
原文摘要 · Abstract (English)
LLMs have recently achieved strong results on formal proving benchmarks. However, existing evaluations remain heavily concentrated on competition-style problems and often fail to capture how models behave on longer, more dependency-rich mathematical developments. We introduce TheoremBench, a Lean4 benchmark designed to evaluate theorem provers beyond contest settings. The benchmark is built from nearly one hundred classical theorems and is released in two complementary forms: a plain main version containing one target theorem per instance, and a premised version that expands each theorem into a structured family of related proving tasks consisting of the main theorem together with automatically extracted supporting subtheorems. This design enables evaluation of not only whether the final theorem was proved from scratch, but also of partial progress through the internal proof structure of a theorem. Our experiments show that explicit premises substantially improve performance for Lean4-capable prover models. To provide a comprehensive evaluation, we introduce theorem-level coverage and token-efficiency metrics that expose qualitative differences in proof behavior. The results show that current provers remain strongly biased toward easy subtheorems and often solve theorems through long and inefficient tactic traces rather than compact proof plans. TheoremBench therefore provides a more fine-grained view of formal reasoning ability and highlights the importance of structural benchmark design for evaluating Lean4 theorem provers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。