大模型模仿逻辑推理，但成本惊人；新方法降本增效。
Large Language Models Imitate Logical Reasoning, but at what Cost?
- 用小模型将问题转为标准形式，再交由Z3求解器验证。
- 计算量降低90%以上，准确率接近100%。
- 适合关注推理效率与成本的开发者与研究者。
我们开展了一项历时十八个月的纵向研究，评估前沿大模型在逻辑推理能力上的演变。通过在PrOntoQA数据集上测试2023年12月、2024年9月和2025年6月三个阶段的三款领先模型，发现从2023到2024年的性能提升主要归因于隐藏的链式思维提示（Chain of Thought prompting），而2024到2025年则因引入思考型模型实现显著进步。随后提出一种神经符号架构：使用参数少于150亿的小型语言模型将问题转化为标准化形式，并解析为程序供Z3 SMT求解器求解，以判断查询的可满足性。报告了开源模型的提示与完成令牌数及计算成本（FLOPs）。该方法大幅降低计算开销，同时保持近似完美的性能。实验表明，所有情况下推理所需FLOPs约为活跃参数与总令牌数乘积的两倍，误差控制在10%以内。
原文摘要 · Abstract (English)
We present a longitudinal study which evaluates the reasoning capability of frontier Large Language Models over an eighteen month period. We measured the accuracy of three leading models from December 2023, September 2024 and June 2025 on true or false questions from the PrOntoQA dataset and their faithfulness to reasoning strategies provided through in-context learning. The improvement in performance from 2023 to 2024 can be attributed to hidden Chain of Thought prompting. The introduction of thinking models allowed for significant improvement in model performance between 2024 and 2025. We then present a neuro-symbolic architecture which uses LLMs of less than 15 billion parameters to translate the problems into a standardised form. We then parse the standardised forms of the problems into a program to be solved by Z3, an SMT solver, to determine the satisfiability of the query. We report the number of prompt and completion tokens as well as the computational cost in FLOPs for open source models. The neuro-symbolic approach significantly reduces the computational cost while maintaining near perfect performance. The common approximation that the number of inference FLOPs is double the product of the active parameters and total tokens was accurate within 10\% for all experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。