arXiv:2412.15177cs.AIcs.CL2024-12被引 3

用批判性问题引导大模型自我纠错,提升逻辑推理能力。

Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying

  • 引入论证理论中的批判性问题,让大模型自检推理过程。
  • 在多任务测试中,性能显著优于基线和思维链方法。
  • 适合需要高可靠性推理的应用场景,如数学证明与法律分析。

尽管人工智能研究取得快速进展,当前最先进的大语言模型(LLMs)在逻辑与数学推理方面仍表现不佳。这表明它们更像高级的数据模式识别器,难以泛化到未见过或与训练数据差异较大的推理任务。本文借鉴论证理论中的批判性问题,特别是图尔敏论证模型,提出一种通过追问模型推理依据来改进其推理能力的方法。该方法使模型能识别潜在逻辑错误并提前修正,从而输出更可靠的结论。核心思想源于有效论证的标准:结论成立当且仅当其由可接受前提推出;在现实不完全信息下,则以‘未被证伪’为有效标准。实验在MT-Bench Reasoning和Math任务上对多种LLM进行评估,结果表明该方法显著优于基线及思维链(CoT)实现。

原文摘要 · Abstract (English)

Studies have underscored how, regardless of the recent breakthrough and swift advances in AI research, even state-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning. The results seem to suggest that LLMs still work as (highly advanced) data pattern identifiers, scoring poorly when attempting to generalise and solve reasoning problems the models have never previously seen or that are not close to samples presented in their training data. To address this compelling concern, this paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation. We show that employing these critical questions can improve the reasoning capabilities of LLMs. By probing the rationale behind the models' reasoning process, the LLM can assess whether some logical mistake is occurring and correct it before providing the final reply to the user prompt. The underlying idea is drawn from the gold standard of any valid argumentative procedure: the conclusion is valid if it is entailed by accepted premises. Or, to paraphrase such Aristotelian principle in a real-world approximation, characterised by incomplete information and presumptive logic, the conclusion is valid if not proved otherwise. This approach successfully steers the models' output through a reasoning pipeline, resulting in better performance against the baseline and its Chain-of-Thought (CoT) implementation. To this end, an extensive evaluation of the proposed approach on the MT-Bench Reasoning and Math tasks across a range of LLMs is provided.

逻辑推理大模型批判性思维

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。