用形式化逻辑符号提升小模型的三段论推理能力。
Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities
- 用形式化知识表示法改写输入，让小模型更好理解逻辑关系。
- 在零样本场景下，新方法使模型准确率提升至62.3%。
- 开源工具可自动生成逻辑语句，适合研究逻辑推理的学者。
语言模型在三段论等逻辑任务上表现不佳。研究表明，知识表示（KR）对信息表达至关重要。本文通过扩展FOLIO和P-FOLIO数据集，研究不同形式化KR符号对三段论推理的影响。在小语言模型的监督微调（SFT）与零样本（ZS）设置中，采用特定符号表示的输入性能媲美自然语言，且推理速度更快。我们提出一种三段论分类方法（SEF），并用于增强零样本提示中的逻辑定义，显著提升小模型推理效果。本文开源了首个自动化生成三段论逻辑语句及定义类别（SEF）的Python库——通用逻辑语法构建框架（CLGC）。
原文摘要 · Abstract (English)
Language models (LMs) struggle with logical tasks like reasoning on syllogisms. It has been shown that Knowledge Representation (KR) plays a crucial role in expressing input information to help models solve tasks. This observation motivates our study of the impact of different formal KR notations on syllogistic reasoning by extending the FOLIO and P-FOLIO datasets. Our experiments on Small Language Models (SLMs) in Supervised Fine-Tuning (SFT) and Zero-Shot (ZS) settings show that the choice of input notation can yield performances competitive with natural language while enabling faster inference. We also propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions, which boost reasoning in small models. We open-source our framework, Common Logic Grammar Construction (CLGC), as the first Python library for automatically generating syllogisms in KR notations and defining their SEF categories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。