用逻辑等价性检测大模型推理缺陷，发现传统评测漏掉的隐藏问题。
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
- 基于一阶逻辑生成语义不变的测试用例，通过一致性检查发现错误。
- 六款主流模型中均暴露传统评测未发现的推理缺陷。
- 适合评估模型在逻辑变换下的鲁棒性，尤其关注符号和结论变化。
大型语言模型（LLMs）在逻辑推理基准上表现强劲，但其可靠性仍不确定。现有评估依赖静态基准，无法检验逻辑等价变换下的鲁棒性，常高估推理能力。我们提出LGMT（Logic-Grounded Metamorphic Testing），一种无需人工标注的框架，利用一阶逻辑（FOL）评估LLM推理。通过形式化逻辑等价性推导变换关系，构建语义不变的测试用例，并通过跨用例一致性检查识别推理缺陷。在六款先进LLM上的实验表明，LGMT揭示了传统基于参考答案的评估所遗漏的显著隐藏缺陷。进一步发现，模型对符号级和结论级变化尤为敏感，而少样本思维链（Few-shot CoT）提示仅部分缓解此问题。结果表明，LLM评估应从孤立正确性转向逻辑不变性下的鲁棒性。LGMT提供了一种原则性强且可扩展的推理失败诊断方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations rely on static benchmarks, which fail to assess robustness under logically equivalent transformations and often overestimate reasoning capability. We propose LGMT (Logic-Grounded Metamorphic Testing), an oracle-free framework that leverages first-order logic (FOL) to evaluate LLM reasoning. By deriving metamorphic relations from formal logical equivalences, LGMT constructs semantically invariant test cases and detects reasoning defects through cross-case consistency checking. Experiments on six state-of-the-art LLMs show that LGMT exposes substantial hidden defects missed by traditional reference-based evaluations. We further find that models are particularly sensitive to symbol-level and conclusion-level variations, and that advanced prompting such as Few-shot CoT only partially mitigates these issues. These results suggest that LLM evaluation should move beyond isolated correctness toward robustness under logical invariance. LGMT provides a principled and scalable approach for diagnosing reasoning failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。