发现并修复了两个重要逻辑翻译数据集的标注错误,显著提升模型评估准确性。
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

- 通过人工审核发现39%的FOLIO和36%的MALLS标注有误
- 修正标注后,大模型准确率最高提升22个百分点
- 用LLM辅助筛选高风险样本,减少70%以上人工审查量
NL-to-FOL(自然语言转一阶逻辑)的准确性是神经符号人工智能和自然语言推理的基础,但现有数据集从未经过严格审计。本文对FOLIO验证集和部分MALLS测试集进行系统人工检查,发现约39%和36%的条目存在错误的一阶逻辑形式化(即真实标签),另有16.4%和48%的自然语言句子表述模糊,以及8.4%的FOLIO存在错误的自然语言推理标签。我们发布了经人工校验的修正版真实标签,结果显示,使用修正标签重新评估三个主流大模型(Gemma 4 31B-it、Qwen3-30B-A3B、GPT-4o-mini)时,准确率提升9至22个百分点。基于此,我们提出一种基于LLM的框架,帮助人类高效识别高风险样本。实证表明,仅需审查不足24%的样本即可达到90%的数据集准确率,远低于无指导审查所需的70%以上。所有验证标注与框架代码均已公开。
原文摘要 · Abstract (English)
Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited. Our first contribution is to present a systematic human inspection of the validation split of \textsf{FOLIO} and a subset of \textsf{MALLS} test instances, finding that approximately 39% and 36% of entries, respectively, contain incorrect FOL formalizations (i.e., ground truth labels), with additional rates of ambiguous NL sentences (16.4% and 48%) and incorrect NLI labels in \textsf{FOLIO} (8.4%). Our second contribution is to develop and release corrected ground truths for such datasets, showing that annotation errors distort model evaluation on a reference benchmark task: testing three state-of-the-art LLMs (Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini) with the corrected ground truths yields accuracy gains from +9 to +22 percentage points. Motivated by these findings, we propose an LLM-based framework to support humans in manual reviewing NL-to-FOL datasets. By directing reviewers toward the most error-prone instances, we empirically show that it is possible to achieve 90% dataset accuracy after reviewing fewer than 24% of instances, compared to over 70% required by unguided review. We release all human-verified annotations and the code for our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。