arXiv:2502.02362cs.CL2025-02ICML被引 28

通过提取每步推理前提,提升大模型数学推理错误识别能力

Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs

  • 将线性推理链重构为带前提链接的有向无环图
  • 开源模型在复杂链中前提识别召回率达90%
  • 结合前提验证后,错误识别准确率提升6%至16%

Chain-of-Thought (CoT) 提示能增强大语言模型(LLMs)的数学推理能力,但因模型输出冗长,推理链难以验证,且步骤间依赖关系较远时问题难追踪。数学推理中每一步可由前序少数前提推导得出。本文提出框架,为每步识别前提,将传统线性推理链重构为前提增强推理链(PARC),形成以步骤为节点、前提链接为边的有向无环图。基于自建数据集PERL(Premises and ERrors identification in LLMs)的实验表明,LLMs能可靠识别复杂推理链中的前提,即使开源模型前提识别召回率也达90%。同时,结合前提进行逐步验证,错误识别准确率提升6%至16%绝对值。研究证明前提中心表示对复杂任务的可靠性提升具有价值,并为改进基于LLM的推理评估开辟新路径。

原文摘要 · Abstract (English)

Chain-of-Thought (CoT) prompting enhances mathematical reasoning in large language models (LLMs) by enabling detailed step-by-step solutions. However, due to the verbosity of LLMs, the resulting reasoning chains can be long, making it harder to verify the reasoning steps and trace issues resulting from dependencies between the steps that may be farther away in the sequence of steps. Importantly, mathematical reasoning allows each step to be derived from a small set of premises, which are a subset of the preceding steps in the reasoning chain. In this paper, we present a framework that identifies the premises for each step, to improve the evaluation of reasoning. We restructure conventional linear reasoning chains into Premise Augmented Reasoning Chains (PARC) by introducing premise links, resulting in a directed acyclic graph where the nodes are the steps and the edges are the premise links. Through experiments with a PARC-based dataset that we built, namely PERL (Premises and ERrors identification in LLMs), we demonstrate that LLMs can reliably identify premises within complex reasoning chains. In particular, even open-source LLMs achieve 90% recall in premise identification. We also show that PARC helps to identify errors in reasoning chains more reliably. The accuracy of error identification improves by 6% to 16% absolute when step-by-step verification is carried out in PARC under the premises. Our findings highlight the utility of premise-centric representations in addressing complex problem-solving tasks and open new avenues for improving the reliability of LLM-based reasoning evaluations.

数学推理错误检测推理链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。