arXiv:2505.05758cs.AIcs.LO2025-05NeurIPS被引 44

用智能体协作修复大模型生成的数学证明,效率和准确率大幅提升。

APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning

论文配图:APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning
图 1 · 摘自论文原文
  • 构建自动化框架,让大模型与形式化语言系统协同修正证明错误。
  • 在miniF2F上达到84.9%准确率,采样次数低于100次。
  • 适用于小参数模型,显著提升通用模型的定理证明能力。

形式化推理与自动定理证明是机器学习中的挑战性领域,要求机器使用如Lean等形式语言证明数学定理。虽然形式验证系统能瞬间判断证明正确性,但利用大语言模型(LLMs)生成完全正确的形式化证明仍极困难。现有方法通常需多次提示(达数千次)直至生成可通过验证的证明。本文提出APOLLO(Automated PrOof repair via LLM and Lean cOllaboration),一个模块化、模型无关的智能体框架,结合Lean编译器与大模型推理能力,在低令牌与采样预算下实现更优的证明生成效果。Apollo驱动全自动流程:大模型生成证明,多个智能体分析并修复语法错误,利用Lean识别错误,定位失败子引理,调用自动化求解器,并对剩余目标以低top-K预算调用大模型。修复后的子证明重新组合并重验,最多迭代用户设定次数。在miniF2F基准上,亚80亿参数模型达到84.9%的新高准确率,采样预算低于100。同时,Goedel-Prover-SFT准确率提升至65.6%,采样复杂度从25,600降至数百。通用模型(o3-mini, o4-mini)准确率从3-7%跃升至40%以上。结果表明,针对性的编译器引导修复可极大提升效率与正确性,为可扩展自动定理证明提供通用范式。

原文摘要 · Abstract (English)

Formal reasoning and automated theorem proving constitute a challenging subfield of machine learning, in which machines are tasked with proving mathematical theorems using formal languages like Lean. A formal verification system can check whether a formal proof is correct or not almost instantaneously, but generating a completely correct formal proof with large language models (LLMs) remains a formidable task. The usual approach in the literature is to prompt the LLM many times (up to several thousands) until one of the generated proofs passes the verification system. In this work, we present APOLLO (Automated PrOof repair viaLLM and Lean cOllaboration), a modular, model-agnostic agentic framework that combines the strengths of the Lean compiler with an LLM's reasoning abilities to achieve better proof-generation results at a low token and sampling budgets. Apollo directs a fully automated process in which the LLM generates proofs for theorems, a set of agents analyze the proofs, fix the syntax errors, identify the mistakes in the proofs using Lean, isolate failing sub-lemmas, utilize automated solvers, and invoke an LLM on each remaining goal with a low top-K budget. The repaired sub-proofs are recombined and reverified, iterating up to a user-controlled maximum number of attempts. On the miniF2F benchmark, we establish a new state-of-the-art accuracy of 84.9% among sub 8B-parameter models (as of August 2025) while keeping the sampling budget below one hundred. Moreover, Apollo raises the state-of-the-art accuracy for Goedel-Prover-SFT to 65.6% while cutting sample complexity from 25,600 to a few hundred. General-purpose models (o3-mini, o4-mini) jump from 3-7% to over 40% accuracy. Our results demonstrate that targeted, compiler-guided repair of LLM outputs yields dramatic gains in both efficiency and correctness, suggesting a general paradigm for scalable automated theorem proving.

定理证明大模型智能体形式验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。