arXiv:2410.15748cs.AI2024-10ICLR被引 4

通过符号变异生成百万级新定理,显著提升形式化证明模型性能。

Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation

论文配图:Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation
图 1 · 摘自论文原文
  • 用符号替换方式从已有定理生成新定理,实现数据量十倍增长。
  • 在Leandojo上提升4.70%性能,在miniF2F上提升2.47%。
  • 适合研究形式化证明与大模型训练的开发者参考。

形式化证明对专家而言仍具挑战性。尽管神经定理证明(NTP)取得进展,但互联网上的形式化语料远少于通用文本,导致数据稀缺问题严重。本文提出Alchemy框架,通过符号变异构建形式定理。针对Mathlib中的每个候选定理,识别可调用的定理,将其命题中的项替换为等价形式或前件,从而生成新定理。该方法使Mathlib定理数量从11万增至600万。在扩增语料上进行持续预训练和监督微调后,实验表明:在Leandojo基准上获得4.70%的绝对性能提升;在分布外的miniF2F基准上,基于合成数据实现2.47%的性能增益。此外,我们对合成数据构成与训练范式进行了全面分析,为构建强定理证明器提供指导。

原文摘要 · Abstract (English)

Formal proofs are challenging to write even for experienced experts. Recent progress in Neural Theorem Proving (NTP) shows promise in expediting this process. However, the formal corpora available on the Internet are limited compared to the general text, posing a significant data scarcity challenge for NTP. To address this issue, this work proposes Alchemy, a general framework for data synthesis that constructs formal theorems through symbolic mutation. Specifically, for each candidate theorem in Mathlib, we identify all invocable theorems that can be used to rewrite or apply to it. Subsequently, we mutate the candidate theorem by replacing the corresponding term in the statement with its equivalent form or antecedent. As a result, our method increases the number of theorems in Mathlib by an order of magnitude, from 110k to 6M. Furthermore, we perform continual pretraining and supervised finetuning on this augmented corpus for large language models. Experimental results demonstrate the effectiveness of our approach, achieving a 4.70% absolute performance improvement on Leandojo benchmark. Additionally, our approach achieves a 2.47% absolute performance gain on the out-of-distribution miniF2F benchmark based on the synthetic data.To provide further insights, we conduct a comprehensive analysis of synthetic data composition and the training paradigm, offering valuable guidance for developing a strong theorem prover.

定理证明数据增强符号变异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。