测试大模型在混乱数学证明下的纠错能力,发现现有方法极易出错。
Evaluating the Robustness of Proof Autoformalization in Lean 4

- 设计两类干扰:风格改写和局部修改,检验模型稳定性。
- 7个主流模型在风格改写下表现不稳,局部修改时多错误重置或误推。
- 首次系统评估数学证明自动形式化的鲁棒性,适合形式化研究者参考。
证明自动形式化旨在将自然语言撰写的数学非正式证明转化为形式语言(如 Lean 4)中的严格证明。已有工作基于大语言模型构建了此类模型,但现有评估通常仅针对结构良好的非正式证明。我们认为,一个稳健的自动形式化器必须在非理想证明下仍保持忠实,因此首次系统研究该任务的鲁棒性。我们提出两类扰动:全局扰动(改写证明风格)要求形式化结果一致;局部扰动(修改数值、符号或步骤)要求输出真实反映变化,而非恢复原状或自行推断。我们在 miniF2F 和 MATH-500 数据集上构建包含两类扰动的基准,并自动衡量模型在全局扰动下的正确性稳定性与局部扰动下的忠实度。评估的7个近期模型均对全局扰动敏感,且多数无法在局部扰动下保持忠实。代码与数据已开源。
原文摘要 · Abstract (English)
Proof autoformalization aims to translate a mathematical informal proof written in natural language into a formal proof in a formal language such as Lean~4. Several works have developed LLM-based models for proof autoformalization. However, existing evaluations have typically focused on translating well-formed informal proofs from curated datasets. We argue that a robust proof autoformalizer must remain faithful even for informal proofs that diverge from these idealized ones, and we present the first study on the robustness of proof autoformalization models. We formulate two categories of perturbations and evaluate robustness under each: a global perturbation paraphrases the informal proof in a different style, under which the formalization should remain consistent; a local perturbation alters a value, symbol, or proof step, possibly in a counterfactual way, and a robust formalization should faithfully reflect the perturbation rather than reverting to the original one or inferring a different one on its own. We build a benchmark with both perturbations on miniF2F and MATH-500, and automatically measure how stable a proof autoformalization's correctness is under global perturbations and how faithfully its output reflects local perturbations. We evaluate seven recent models, all of which are sensitive to global perturbations and mostly fail to remain faithful under local perturbations. Code and data are available via https://github.com/ucr-rai/robust-proof-autoformalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。