测试大模型在最基础归纳推理任务上的表现，发现其能力严重不足。
InductionBench: LLMs Fail in the Simplest Complexity Class
- 构建新基准InductionBench，聚焦从数据中归纳规则的推理能力。
- 即使最强模型也无法掌握最简单的子正则层次函数类。
- 适合关注大模型本质推理缺陷的研究者阅读。
大型语言模型（LLMs）在推理能力上取得显著进步，许多现有基准已被o1和o3等模型部分或完全解决。然而，这些基准大多强调演绎推理，如数学和编程任务，其规则（如数学公理或编程语法）明确，模型可据此规划并应用规则求解。相比之下，归纳推理——即从观测数据中推断潜在规律——仍鲜受关注。此类过程是科学发现的核心，使研究者能从实证观察中提炼普遍原则。为评估LLMs是否具备此能力，我们提出InductionBench，一个专用于评估归纳推理能力的新基准。实验结果表明，即便是当前最先进的模型，也难以掌握子正则层次中最简单的复杂度类，暴露出当前大模型在归纳推理能力上的明显短板。代码与数据已公开于https://github.com/Wenyueh/inductive_reasoning_benchmark。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable improvements in reasoning and many existing benchmarks have been addressed by models such as o1 and o3 either fully or partially. However, a majority of these benchmarks emphasize deductive reasoning, including mathematical and coding tasks in which rules such as mathematical axioms or programming syntax are clearly defined, based on which LLMs can plan and apply these rules to arrive at a solution. In contrast, inductive reasoning, where one infers the underlying rules from observed data, remains less explored. Such inductive processes lie at the heart of scientific discovery, as they enable researchers to extract general principles from empirical observations. To assess whether LLMs possess this capacity, we introduce InductionBench, a new benchmark designed to evaluate the inductive reasoning ability of LLMs. Our experimental findings reveal that even the most advanced models available struggle to master the simplest complexity classes within the subregular hierarchy of functions, highlighting a notable deficiency in current LLMs' inductive reasoning capabilities. Coda and data are available https://github.com/Wenyueh/inductive_reasoning_benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。