提出PEA框架，让大模型更高效解决计算类推理题
PEA: Enhancing LLM Performance on Computational-Reasoning Tasks
- 将复杂推理拆解为谓词、枚举和聚合三部分，用大模型生成可执行程序
- 在布尔可满足性等任务上平均准确率提升约50%，且执行更高效
- 适合想提升大模型算术与逻辑推理能力的研究者和开发者
大型语言模型在多个领域展现出强大能力，促使人们探索其作为通用推理引擎的潜力。尽管近期研究尝试通过推理时计算来提升模型在复杂问题上的表现，但现有工作缺乏对推理任务复杂性的正式描述框架。本文提出谓词-枚举-聚合（Predicate-Enumeration-Aggregation, PEA）框架，用于描述并求解一类重要的推理任务——计算推理问题。该框架将问题分解为谓词和枚举组件，利用大模型根据指定谓词、枚举规则和聚合规则生成程序，再执行这些程序以获得计算任务的解。我们在布尔可满足性问题、24点游戏及规划问题等基准任务上验证了该框架的有效性。实证结果表明，PEA显著提升了基础模型在基准计算问题上的性能，平均准确率提升约50%，同时兼具更高的效率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have exhibited remarkable capabilities across diverse domains, prompting investigations into their potential as generic reasoning engines. While recent studies have explored inference-time computation to enhance model performance on complex problems, current research lacks a formal framework to characterize the complexity of reasoning tasks. This study introduces the Predicate-Enumeration-Aggregation (PEA) framework, a formal approach to describe and solve a class of important reasoning tasks termed computational reasoning problems. The PEA framework decomposes these problems into predicate and enumeration components, using LLMs to synthesize programs based on specified predicates, enumeration, and aggregation rules. These synthesized programs are then executed to obtain solutions to the computational tasks. We demonstrate the framework's efficacy on benchmark tasks including Boolean satisfiability problems, game of $24$, and planning problems. Empirical evaluation reveals that PEA substantially enhances the performance of underlying models on benchmark computational problems, yielding an average accuracy improvement of approximately $50\%$, coupled with increased efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。