arXiv:2606.31002cs.AIcs.CL2026-06被引 1

评估自然语言转Lean形式化时,发现编译通过不等于语义忠实。

Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

论文配图:Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
图 1 · 摘自论文原文
  • 设计多维度评估协议,融合编译、模型判别与专家校准
  • 89.5%编译通过率但仅60.5%语义忠实,差距达29.0点
  • 强调应分开报告形式有效性、证明能力与语义忠实性

定理证明基准测试通常在固定的形式陈述上评估证明搜索,但在自然语言转Lean形式化任务中,需自动生成形式陈述。此时编译仅是有效性检查:一个可类型检查的Lean声明可能遗漏假设、改变域或表达空命题。本文将忠实形式化视为评估问题与瓶颈归因问题。在包含400个研究生级条目的基准(涵盖实分析、复分析、拓扑与代数)上,我们的协议结合了Lean编译、跨模型语义判别与人工专家校准。结果表明,与仅依赖编译率不同,全工具增强代理的编译通过率为89.5%,但共识忠实率仅为60.5%,暴露29.0点的编译通过但共识不忠实差距。针对性的人工审计支持该度量作为保守决策边界:在可用案例级审计中,96.0%的共识阳性输出经人工确认为忠实,而82.4%的编译通过但共识阴性输出被确认为语义失败。在此度量下,现有单次形式化模型与以证明为导向的Lean模型仍表现较低,提示应分别报告形式有效性、以证明为导向的Lean能力与忠实陈述生成。随后我们采用完整的$2^3$因子设计,分解形式化流程中三个常见干预措施:参数化专家起草、Mathlib/上下文检索与Lean展开反馈。展开反馈是最大的有效性干预,但也暴露更大的编译通过但语义失败子集;检索主要提升锚定与选择性;而微调后的起草在反馈和锚定可用后基本可被替代。

原文摘要 · Abstract (English)

Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.

形式化自然语言验证数学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。