arXiv:2501.08613cs.CL2025-01EMNLP被引 1

评估逻辑表达式相似度指标的敏感性与对齐度,提升大模型推理的可信度。

Assessing the Sensitivity and Alignment of FOL Closeness Metrics

  • 通过扰动操作符和文本构建测试用例,评估各类指标对差异的敏感度。
  • BLEU对文本扰动过度敏感,Smatch++受结构变化影响,而BertScore与人类判断更一致。
  • 融合多个指标可兼顾敏感性与鲁棒性,适合评估大模型生成的逻辑表达式。

工具增强型大语言模型(LLM)在解决逻辑推理问题时,常将自然语言(NL)转化为一阶逻辑(FOL)并借助外部定理证明器。然而,由于缺乏可靠的生成与真实FOL对比评价指标,生成结果的正确性往往难以验证。本文系统评估了现有基于自然语言、一阶逻辑和图结构的度量方法对采样FOL与真实FOL之间差异的敏感性,并测量其与强模型作为评判者(LLM as-a-judge)的一致性。我们首先对真实FOL进行操作符和文本扰动以测试指标敏感性;再通过与LLM判断对比,评估指标鲁棒性。实验发现:n-gram类指标BLEU对文本扰动过于敏感;操作符扰动影响结构度量指标Smatch++及特定操作符相关的FOL度量;而BertScore与LLM判断有更高一致性,凸显语义评估的重要性。此外,组合多个指标相比单一指标,在敏感性与鲁棒性上均有提升。

原文摘要 · Abstract (English)

The recent successful paradigm of solving logical reasoning problems with tool-augmented large language models (LLMs) leverages translation of natural language (NL) statements into First-Order Logic~(FOL) and external theorem provers. However, the correctness of FOL statements, comprising operators and text, often go unverified due to the lack of a reliable evaluation metric for comparing generated and ground-truth FOLs. In this paper, we conduct a comprehensive study on the sensitivity of existing NL-, FOL-, and graph-based metrics to capture differences between a sampled FOL and its corresponding ground-truth. We then measure the alignment between a metric-based ranking of FOL outputs and a strong LLM as-a-judge. To do this, we first apply operator and text-based perturbations to ground-truth FOL statements to assess metric sensitivity. We then evaluate metric robustness by comparing the metrics against LLMs judgment. Our empirical findings highlight a clear oversensitivity in the n-gram metric BLEU for text perturbations. The operator perturbation affects the semantic graph metric Smatch++ for structural changes, and the FOL metric for specific operator changes. We observe a closer alignment between BertScore and LLM judgement, proving the importance of semantic evaluation. Additionally, we show that combining metrics enhances both robustness and sensitivity compared to using individual metrics.

逻辑推理大模型评估语义匹配指标设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。