arXiv:2605.12524cs.LOcs.AI2026-05被引 1

用可机器验证的证明测试大模型推理能力,发现其仍有重大短板。

Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism

论文配图:Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism
图 1 · 摘自论文原文
  • 设计最小形式化语言NDL,让模型写出可自动验证的证明过程。
  • 前沿模型在基础任务表现良好,但复杂组合推理仍无法解决。
  • 提出新评估框架与稳定性指标,适合研究模型真实推理能力。

我们提出ProofGrid,一个通过可机器验证的证明而非仅最终答案来评估大模型推理能力的基准套件。ProofGrid包含15个任务,涵盖证明撰写、检查、掩码和补全,使用最小形式化符号表达,特别是紧凑的自然演绎语言NDL,便于短提示输入并支持精确、可审计的验证。该方法实现机械式、可复现且细粒度的评估,避免依赖人工或模型判断。任务覆盖从基础推理到结构复杂的挑战性任务,现有模型均未完全解决,同时尽量减少对领域知识、求解器调用和长上下文的依赖。我们还构建了对比评估框架,从表示方式、验证保障和推理深度三方面定位ProofGrid。方法上,引入可容忍微小表面差异但能定位首次实质性错误的仪器化验证流程,提升测量精度,分离证明规划与低级执行噪声。基于此流程,评估了多款开源与专有模型。结果显示,前沿模型在基础任务上表现良好,但在需要全局组合推理或低层证明生成的难题上仍有显著差距。我们识别出‘认知不稳定性’现象:模型生成错误证明,却能正确否定孤立的局部推论,并提出‘认知稳定性指数’进行量化。最后,结合2PL IRT分析、Wright图及基于费舍尔信息的归一化任务区分度指标,补充准确率评估。

原文摘要 · Abstract (English)

We introduce ProofGrid, a benchmark suite for evaluating LLM reasoning through machine-checkable proofs rather than final answers alone. ProofGrid contains 15 tasks spanning proof writing, proof checking, proof masking, and proof gap-filling. Tasks are expressed in minimal formal notation, especially NDL, a compact natural-deduction language that fits in short prompts and supports precise, auditable verification. This yields mechanical, reproducible, and fine-grained evaluation rather than judgments by humans or LLMs. ProofGrid covers a calibrated difficulty spectrum, from foundational reasoning tests to structurally rich challenge tasks that no current model solves, while minimizing reliance on domain knowledge, solver delegation, and long-context artifacts. We also develop a comparative framework for reasoning benchmarks and use it to situate ProofGrid relative to existing work in terms of representation, verification guarantees, and reasoning depth. Methodologically, we introduce an instrumented proof-checking pipeline that tolerates minor surface deviations while locating the first substantive reasoning failure, improving measurement resolution and separating proof planning from low-level execution noise. Using this pipeline, we evaluate a broad range of open and proprietary models. Results show rapid progress but substantial remaining limits: frontier models perform well on several foundational tasks, yet difficult tasks, especially those requiring global combinatorial reasoning or low-level proof synthesis, remain far from solved. We also identify epistemic instability, where models generate flawed proofs yet correctly reject those local inferences in isolation, and formalize this with an Epistemic Stability Index. Finally, we complement accuracy with 2PL IRT analyses, Wright maps, and a normalized task-discrimination measure based on Fisher information.

大模型推理形式化证明评估基准认知稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。