arXiv:2607.23425cs.SEcs.AI2026-07

用执行验证替代传统评估,更真实地衡量自然语言转TLA+规范的正确性。

TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA Specification Generation

论文配图:TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA Specification Generation
图 1 · 摘自论文原文
  • 以模型检查执行结果作为正确性标准,而非仅看语法或相似度。
  • 同一模型输出在不同评估条件下正确率从1.7%到18.7%波动,跨度达十一倍。
  • 适合关注形式化验证、LLM生成质量评估的研究者与实践者。

大型语言模型越来越多地从自然语言描述生成TLA⁺形式规范,但现有评估方式难以衡量实际正确性:以往资源仅依据与参考答案的相似性或语法可解析性评分。本文提出TLA⁺-Bench,一个基于执行验证的数据集与基准。每个标准规范都配备一个配置,由TLA⁺模型检查器遍历全部可达状态空间,精确判断规范是否满足该配置所命名的性质。数据集包含403个经过模型检查的黄金规范和897个仅通过语法检查的银色规范,来自13个公共代码库,涵盖两个提供方的两种风格的模型撰写描述,并附有难度与类别标签。主要发现是关于评估本身:一个精确的评判标准并非单一数值,而是一个范围。仅改变以往基准未说明的评分选择,在同一组模型输出上,正确率在1.7%至10.0%间变动;若加入接口提供选项(即告知模型配置名称),正确率范围扩大至1.7%至18.7%,跨度达十一倍。我们称此为‘正确性包络’,并测量其上下界。包络内的发现稳定:所有模型写出有效TLA⁺远多于正确规范——最强模型默认正确率为16%,接口提示下升至26%;开源模型最多仅1%;正确率随任务难度急剧下降。

原文摘要 · Abstract (English)

Large language models increasingly write TLA$^{+}$ formal specifications from natural-language descriptions, but progress is hard to measure: existing resources grade by resemblance to a reference or by whether the output parses, neither of which shows correctness. We present TLA$^{+}$-Bench, a dataset and benchmark that grades by execution. Every gold specification ships a configuration the TLA$^{+}$ model checker runs over the full reachable state space, deciding exactly whether the specification holds the properties that configuration names. The dataset holds 403 model-checked gold and 897 parse-only silver specifications from 13 public repositories, subsumes prior TLA$^{+}$ generation data, and carries four model-written descriptions in two styles from two providers, with difficulty and category labels. Our main finding is about measurement itself: an exact oracle gives not one correctness number but a range. Varying only the grading choices earlier benchmarks leave unstated, on one fixed set of model outputs, the correct rate moves sixfold, from 10.0\% to 1.7\%; adding the interface-supply choice, where the model is told the configuration's names, widens the range to elevenfold, from 18.7\% to 1.7\%. We call this range the correctness envelope and measure each of its bounds. The findings inside it are stable. Every model writes valid TLA$^{+}$ far more often than correct TLA$^{+}$: the strongest is correct 16\% of the time by default and 26\% when given the interface names, open models at most 1\%, and correctness falls sharply with difficulty.

形式化验证LLM评估TLA+生成质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。