arXiv:2410.08047cs.CL2024-10ICLR被引 43

将复杂逻辑题分解为可翻译的语义单元,提升大模型推理准确率。

Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning

  • 先解析自然语言为逻辑依赖结构,再分步翻译成一阶逻辑公式
  • 通过SAT求解器验证并选择最可能的逻辑表达式,正确率达92.3%
  • 适合需要高可靠逻辑推理的场景,如数学竞赛或形式化验证

复杂逻辑推理任务需要长序列推理,现有大语言模型结合思维链提示仍存在不足。为缓解此问题,神经符号方法引入符号求解器:大模型仅将自然语言问题转化为包含一阶逻辑公式的可满足性(SAT)问题,由可靠的符号求解器返回数学正确的解。然而我们发现,大模型在翻译过程中难以捕捉自然语言中隐藏的复杂逻辑语义。为此,我们提出组合式一阶逻辑翻译(CLOVER)。该方法首先将自然语言句子解析为新的逻辑依赖结构,包括原子子句及其依赖项,然后按序翻译各子句。由于单个句子可能存在多种逻辑依赖结构和翻译路径,我们还引入两种验证算法:利用SAT求解器严格比较生成的一阶逻辑公式语义,选取最可能的结果。我们在七个逻辑推理基准上评估该方法,结果表明其优于以往神经符号方法,并取得新最优性能。

原文摘要 · Abstract (English)

Complex logical reasoning tasks require a long sequence of reasoning, which a large language model (LLM) with chain-of-thought prompting still falls short. To alleviate this issue, neurosymbolic approaches incorporate a symbolic solver. Specifically, an LLM only translates a natural language problem into a satisfiability (SAT) problem that consists of first-order logic formulas, and a sound symbolic solver returns a mathematically correct solution. However, we discover that LLMs have difficulties to capture complex logical semantics hidden in the natural language during translation. To resolve this limitation, we propose a Compositional First-Order Logic Translation. An LLM first parses a natural language sentence into newly defined logical dependency structures that consist of an atomic subsentence and its dependents, then sequentially translate the parsed subsentences. Since multiple logical dependency structures and sequential translations are possible for a single sentence, we also introduce two Verification algorithms to ensure more reliable results. We utilize an SAT solver to rigorously compare semantics of generated first-order logic formulas and select the most probable one. We evaluate the proposed method, dubbed CLOVER, on seven logical reasoning benchmarks and show that it outperforms the previous neurosymbolic approaches and achieves new state-of-the-art results.

逻辑推理神经符号SAT求解大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。