构建动态数学证明基准，评估AI在真实项目中的表现
SorryDB: Can AI Provers Complete Real-World Lean Theorems?
- 从GitHub真实项目提取任务，构建可动态更新的基准
- 1000个任务中Gemini Flash表现最佳，但其他方法各有优势
- 适合关注AI辅助数学形式化与工具实用性的研究者
我们提出SorryDB，一个从GitHub上78个真实形式化项目中动态抽取开放Lean任务的基准。与以往静态基准（多来自竞赛题）不同，该基准能引导工具开发更贴合社区需求，提升数学家可用性，并增强对复杂依赖关系的理解能力。通过持续更新任务流，抱歉数据库缓解了测试集污染问题，提供衡量代理在新形式化项目中贡献能力的稳健指标。我们在一个包含1000个任务的快照上评估了多种方法，包括通用大语言模型、智能体方案和专用符号证明器。结果表明当前方法具有互补性：尽管基于Gemini Flash的智能体表现最优，但并未显著优于其他现成大模型、专用证明器或精心筛选的Lean策略列表。
原文摘要 · Abstract (English)
We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub. Unlike existing static benchmarks, often composed of competition problems, hillclimbing the SorryDB benchmark will yield tools that are aligned to the community needs, more usable by mathematicians, and more capable of understanding complex dependencies. Moreover, by providing a continuously updated stream of tasks, SorryDB mitigates test-set contamination and offers a robust metric for an agent's ability to contribute to novel formal mathematics projects. We evaluate a collection of approaches, including generalist large language models, agentic approaches, and specialized symbolic provers, over a selected snapshot of 1000 tasks from SorryDB. We show that current approaches are complementary: even though an agentic approach based on Gemini Flash is the most performant, it is not strictly better than other off-the-shelf large-language models, specialized provers, or even a curated list of Lean tactics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。