arXiv:2506.07927cs.AIcs.CL2025-06NeurIPS被引 25

用新数据集和评估框架,发现大模型解不等式时推理链脆弱。

Solving Inequality Proofs with Large Language Models

  • 将不等式证明拆为估界和关系预测,可自动验证
  • 顶尖模型在步骤审查下准确率不足10%,比仅看答案差65.5%
  • 揭示当前模型缺乏严谨推理能力,适合研究强化逻辑推理的学者

不等式证明是检验高级推理能力的关键任务,涵盖科学与数学多个领域。现有数据集常稀缺、合成或格式僵化,限制了进展。本文提出一种非正式但可验证的任务形式,将不等式证明分解为两个可自动检查的子任务:边界估计与关系预测。基于此,我们发布IneqMath——一个由专家标注的奥数级不等式数据集,包含训练语料与测试集,并附逐步解题过程与定理注释。同时开发新型LLM作为裁判的评估框架,结合最终答案裁判与四个步骤裁判,用于检测常见推理错误。对29个主流LLM在IneqMath上的系统评估显示:即使顶级模型o1,在步骤审查下的整体准确率也低于10%,相较仅考虑最终答案等价性下降高达65.5%。这一差距暴露了模型推理链条的脆弱性,凸显当前模型在找到答案与构建严谨证明之间的巨大鸿沟。扩大模型规模或增加测试时计算量,对提升证明正确率帮助有限。研究结果指明了未来方向,如定理引导推理与自修正机制。代码与数据已公开于https://ineqmath.github.io/。

原文摘要 · Abstract (English)

Inequality proving, crucial across diverse scientific and mathematical fields, tests advanced reasoning skills such as discovering tight bounds and strategic theorem application. This makes it a distinct, demanding frontier for large language models (LLMs), offering insights beyond general mathematical problem-solving. Progress in this area is hampered by existing datasets that are often scarce, synthetic, or rigidly formal. We address this by proposing an informal yet verifiable task formulation, recasting inequality proving into two automatically checkable subtasks: bound estimation and relation prediction. Building on this, we release IneqMath, an expert-curated dataset of Olympiad-level inequalities, including a test set and training corpus enriched with step-wise solutions and theorem annotations. We also develop a novel LLM-as-judge evaluation framework, combining a final-answer judge with four step-wise judges designed to detect common reasoning flaws. A systematic evaluation of 29 leading LLMs on IneqMath reveals a surprising reality: even top models like o1 achieve less than 10% overall accuracy under step-wise scrutiny; this is a drop of up to 65.5% from their accuracy considering only final answer equivalence. This discrepancy exposes fragile deductive chains and a critical gap for current LLMs between merely finding an answer and constructing a rigorous proof. Scaling model size and increasing test-time computation yield limited gains in overall proof correctness. Instead, our findings highlight promising research directions such as theorem-guided reasoning and self-refinement. Code and data are available at https://ineqmath.github.io/.

不等式证明大模型推理可验证评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。