用理论计算机科学自动生成可验证的数学证明题,解决大模型推理数据稀缺问题。
Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairs
- 基于算法定义自动构造形式化与非形式化对应的问题对
- 大模型在图灵机难题上成功率57.5%,在混合布尔算术题上仅12%
- 适合研究自动化定理证明与大模型推理能力的学者
形式化定理证明(FTP)已成为评估大语言模型推理能力的关键基础,支持大规模数学证明的自动化验证。然而,受限于人工标注成本高和具有验证对应关系的挑战性问题稀缺,进展缓慢。本文提出将理论计算机科学(TCS)作为可扩展的严谨证明问题来源,利用算法定义实现任意数量挑战性定理-证明对的自动化生成。我们在两个TCS领域验证该方法:忙蜂问题(涉及图灵机停机行为的界证明)和混合布尔算术问题(结合逻辑与算术推理)。我们的框架自动合成并行的形式化(Lean4)与非形式化(Markdown)规格,构建了可扩展的已验证证明挑战生成流水线。对前沿模型的评估显示显著差距:DeepSeekProver-V2-671B在忙蜂问题上达到57.5%成功率,但在混合布尔算术问题上仅为12%。结果表明,即使问题验证计算简单,长篇证明生成仍极具挑战,凸显了TCS领域在推动自动化推理研究中的价值。
原文摘要 · Abstract (English)
Formal theorem proving (FTP) has emerged as a critical foundation for evaluating the reasoning capabilities of large language models, enabling automated verification of mathematical proofs at scale. However, progress has been constrained by limited datasets due to the high cost of manual curation and the scarcity of challenging problems with verified formal-informal correspondences. We propose leveraging theoretical computer science (TCS) as a scalable source of rigorous proof problems, where algorithmic definitions enable automated generation of arbitrarily many challenging theorem-proof pairs. We demonstrate this approach on two TCS domains: Busy Beaver problems, which involve proving bounds on Turing machine halting behavior, and Mixed Boolean Arithmetic problems, which combine logical and arithmetic reasoning. Our framework automatically synthesizes problems with parallel formal (Lean4) and informal (Markdown) specifications, creating a scalable pipeline for generating verified proof challenges. Evaluation on frontier models reveals substantial gaps in automated theorem proving: while DeepSeekProver-V2-671B achieves 57.5\% success on Busy Beaver problems, it manages only 12\% on Mixed Boolean Arithmetic problems. These results highlight the difficulty of long-form proof generation even for problems that are computationally easy to verify, demonstrating the value of TCS domains for advancing automated reasoning research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。