构建可验证的自然语言转时序逻辑基准,解决现有方法夸大性能的问题。
Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite
- 提出统一的验证性基准 VLTL-Bench,支持全流程评估
- 包含4个状态空间、数千条自然语言与形式化逻辑对,含样本轨迹
- 分步骤提供真值标签,适合研究可验证翻译的各子模块
现有自然语言到时序逻辑(NL-to-TL)翻译系统的评估表现接近完美,但仅衡量翻译准确率,忽略在新场景中对原子命题的正确定位能力,而这对于公式在具体状态空间中的验证至关重要。多数框架自建数据集,先验已知正确归因,人为抬高性能。本文提出可验证时序逻辑基准(VLTL-Bench),包含4个独立状态空间、数千条自然语言规范及其对应的时序逻辑形式化表达,并附带样本轨迹以验证逻辑公式的正确性。该基准支持端到端评估,同时针对常见分解流程(提升、归因、翻译、验证)提供每一步的真值标签,助力研究者针对性改进与评估。相关数据已发布于Kaggle:https://www.kaggle.com/datasets/dubascudes/vltl-bench。
原文摘要 · Abstract (English)
Empirical evaluation of state-of-the-art natural-language (NL) to temporal-logic (TL) translation systems reveals near-perfect performance on existing benchmarks. However, current studies measure only the accuracy of the translation of NL logic into formal TL, ignoring a system's capacity to ground atomic propositions into new scenarios or environments. This is a critical feature, necessary for the verification of resulting formulas in a concrete state space. Consequently, most NL-to-TL translation frameworks propose their own bespoke dataset in which the correct grounding is known a-priori, inflating performance metrics and neglecting the need for extensible, domain-general systems. In this paper, we introduce the Verifiable Linear Temporal Logic Benchmark ( VLTL-Bench), a unifying benchmark that measures verification and verifiability of automated NL-to-LTL translation. The dataset consists of four unique state spaces and thousands of diverse natural language specifications and corresponding formal specifications in temporal logic. Moreover, the benchmark contains sample traces to validate the temporal logic expressions. While the benchmark directly supports end-to-end evaluation, we observe that many frameworks decompose the process into i) lifting, ii) grounding, iii) translation, and iv) verification. The benchmark provides ground truths after each of these steps to enable researches to improve and evaluate different substeps of the overall problem. To encourage methodologically sound advances in verifiable NL-to-LTL translation approaches, we release VLTL-Bench here: https://www.kaggle.com/datasets/dubascudes/vltl bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。