评估自然语言转Lean形式化时,发现编译通过不等于语义忠实。
Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

- 设计多维度评估协议,融合编译、模型判别与专家校准
- 89.5%编译通过率但仅60.5%语义忠实,差距达29.0点
- 强调应分开报告形式有效性、证明能力与语义忠实性
定理证明基准测试通常在固定的形式陈述上评估证明搜索,但在自然语言转Lean形式化任务中,需自动生成形式陈述。此时编译仅是有效性检查:一个可类型检查的Lean声明可能遗漏假设、改变域或表达空命题。本文将忠实形式化视为评估问题与瓶颈归因问题。在包含400个研究生级条目的基准(涵盖实分析、复分析、拓扑与代数)上,我们的协议结合了Lean编译、跨模型语义判别与人工专家校准。结果表明,与仅依赖编译率不同,全工具增强代理的编译通过率为89.5%,但共识忠实率仅为60.5%,暴露29.0点的编译通过但共识不忠实差距。针对性的人工审计支持该度量作为保守决策边界:在可用案例级审计中,96.0%的共识阳性输出经人工确认为忠实,而82.4%的编译通过但共识阴性输出被确认为语义失败。在此度量下,现有单次形式化模型与以证明为导向的Lean模型仍表现较低,提示应分别报告形式有效性、以证明为导向的Lean能力与忠实陈述生成。随后我们采用完整的$2^3$因子设计,分解形式化流程中三个常见干预措施:参数化专家起草、Mathlib/上下文检索与Lean展开反馈。展开反馈是最大的有效性干预,但也暴露更大的编译通过但语义失败子集;检索主要提升锚定与选择性;而微调后的起草在反馈和锚定可用后基本可被替代。
原文摘要 · Abstract (English)
Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。