用可量化逻辑生成测试用例,精准评测大模型推理能力
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

- 通过一阶逻辑构造可控复杂度的推理任务
- 2880个实例中模型性能随逻辑复杂度上升而下降
- 适合评估大模型在高风险决策中的推理可靠性
大语言模型在推理能力上取得显著进展,尤其在演绎推理方面,这对高风险决策至关重要。然而现有评测基准在逻辑复杂度控制和语义多样性与逻辑一致性平衡上存在不足。为此,我们提出QMFOL,一个基于可量化一元一阶逻辑自动生成推理任务的自动化框架。该框架利用合取与析取模式构建形式化逻辑结构,精确控制推理深度、宽度、标签类型及干扰项。这些结构通过大模型转译为自然语言,并借助外部证明器进行往返验证以确保逻辑一致性。基于此框架,我们构建了包含2880个实例、覆盖960种配置的QMFOLBench基准。对六种大推理模型和两种通用大模型的评估显示,随着逻辑复杂度增加,模型性能下降且计算开销上升;真值类任务表现优于假值或未知类任务,且对语义变化敏感。整体而言,QMFOL提供了一种可扩展、可靠的演绎推理评测方法,支持对现代语言模型推理能力的精准评估。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making. As models improve, evaluation benchmarks should evolve to keep pace. However, existing benchmarks lack fine-grained control over logical complexity and struggle to balance semantic diversity with logical consistency. To address these issues, we propose QMFOL, an automated framework for generating monadic first-order logic reasoning tasks with quantifiable and controllable complexity. It constructs formal logical structures using conjunction and disjunction patterns, enabling precise control over reasoning depth, width, label types, and distractors. These structures are then translated into natural language via LLMs, with logical consistency ensured through round-trip verification using an external prover. Based on our framework, we build QMFOLBench, a benchmark comprising 2880 instances with 960 configurations across diverse logical and semantic dimensions. Evaluations on six large reasoning models (LRMs) and two LLMs show that performance degrades and computational overhead increases with rising logical complexity. Models perform better on True-labeled tasks than on False or Unknown ones, and exhibit sensitivity to semantic variation. Overall, QMFOL offers a scalable and reliable approach for constructing deductive reasoning benchmarks with controllable complexity, enabling more precise evaluation of reasoning capabilities in modern language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。