测试大模型在干扰信息下的推理效率，发现其易被无关信息误导。
Are Language Models Efficient Reasoners? A Perspective from Logic Programming
- 用逻辑编程框架将自然语言证明与最短证明对齐，评估推理效率。
- 引入含干扰公理的数学应用题，模型准确率随干扰增加显著下降。
- 适合关注大模型推理鲁棒性与可解释性的研究者参考。
现代语言模型展现出强大的演绎推理能力，但现有评估侧重正确性，忽视了推理效率这一关键维度。真实推理场景中大量信息无关，高效推理需识别并忽略冗余信息。本文提出一种基于逻辑编程的评估框架，通过将语言模型生成的自然语言证明与逻辑程序执行所得最短证明对齐，量化模型避免无意义推导的能力。我们构建了一个包含数学应用题的数据集，其中注入不同数量的语义重叠程度各异的无关公理。实验发现，即使仅有少量且领域一致的干扰，当前语言模型的准确率也显著下降，其生成的证明频繁出现绕行无关前提的冗余推导。
原文摘要 · Abstract (English)
Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。