根据题目类型自动选符号语言,让大模型逻辑推理更准。
Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning
- 用大模型先选最适合的符号语言再翻译问题。
- 在混合数据集上达到96%准确率,比单一语言高25%。
- 适合需要精准逻辑推理的研究者和开发者。
大语言模型在复杂逻辑推理上仍有困难。现有方法虽提升翻译准确性,但仅关注自然语言与符号语言的语义相似性,忽视了目标符号语言类型的选取这一关键因素。例如,一阶逻辑擅长处理范畴三段论和复杂量词,布尔可满足性形式化则更适合表示约束满足类问题。本文首次提出并验证:不同自然语言逻辑推理题对应最优的符号语言形式化。基于此,我们提出一种自适应选择策略,在翻译前由大模型从一阶逻辑、逻辑编程和布尔可满足性中选出最适目标语言,再进行转换并调用对应逻辑求解器得出答案。实验表明,该方法显著优于固定使用单一符号语言或随机选择的方式。在多个基准数据集的混合测试中,准确率达到96%,较一阶逻辑翻译的最高成绩提升25%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) still struggle with complex logical reasoning. While previous works achieve remarkable improvements, their performance is highly dependent on the correctness of translating natural language (NL) problems into a symbolic language (SL). Though numerous works focusing on improving this translation accuracy, they only consider the similarity between the meaning of SL and NL, overlooking another crucial influencing factor, the selection of the target SL type itself. For example, first-order logic language specializes in logical reasoning with categorical syllogisms and complex quantifiers, while Boolean satisfiability formalism excels at representing constraint satisfaction like partial problems. To our knowledge, this is the first paper to claim and verify that different NL logical reasoning problem corresponds to different optimal SL formalization for translation. Based on this, we propose a methods to improve the logical reasoning performance of LLMs by adaptively selecting the most suitable SL for each problem prior to translation. Specifically, we leverage LLMs to select the target SL among first-order logic, logic programming and Boolean satisfiability and then translate the problem in NL to target SL expressions as well as employ the corresponding logical solver to derive the final answer. Experimental results on benchmarks show that our adaptive selection method significantly outperforms translating all into single SL and randomly selecting the SL. On a mixed dataset of these benchmarks, our approach achieves 96% accuracy, which improving performance by 25% compared to the second highest accuracy from the first-order logic translation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。