arXiv:2608.29411cs.LGcs.SC2026-08

用可计算的描述长度分析发现,OEIS基准主要测记忆而非推理。

Where Induction Runs Out: Description-Length Difficulty and the Memorisation Gap in Integer-Sequence Benchmarks

  • 用最小描述长度评估递推序列的理论发现点,仅与参数量相关。
  • 90%的序列在完整长度下无法拟合递推式,出现理论丢失现象。
  • 语言模型在难序列上不胡编,而是谨慎应对,证明基准偏重记忆。

基于在线整数序列百科全书(OEIS)的整数序列正被广泛用于评估语言模型的数学推理能力。本文通过一个精确可计算的参考学习器——基于P-递归(全纯)递推关系的两段最小描述长度(MDL),对每个序列前缀进行评估。三大发现:第一,MDL难度等价于参数量;理论发现点nd(首次符号假设优于逐字存储的前缀长度)几乎完全由所选算子的阶数与度数的组合可辨识性边界决定,且不受项值大小影响(如斐波那契数列缩放十二个数量级,nd不变),因假设需自编码初值,量级抵消。第二,在20,000个OEIS序列中,89.98%的在某些前缀满足递推关系的序列,在完整长度下均无法拟合,我们称此为“荒野”——归纳获得理论后失去,且永不恢复。第三,对三个语言模型按这些MDL阶段分层评估,推翻了预注册假设:模型在无理论区域不胡编,而是适度规避。自信错误集中于简单阶段,表面能力实际反映的是序列识别而非规则归纳。因此,当前基于OEIS的基准实质测量的是记忆,而MDL提供了一种廉价、无污染的难度信号。

原文摘要 · Abstract (English)

Integer sequences from the On-Line Encyclopedia of Integer Sequences (OEIS) are increasingly used to benchmark mathematical reasoning in language models. We ask what such benchmarks actually measure, using an exactly computable reference learner: two-part minimum description length (MDL) over the class of P-recursive (holonomic) recurrences, evaluated on every prefix of a sequence as terms arrive. Three findings follow. First, MDL difficulty is a parameter count. The discovery point nd, the first prefix length at which a symbolic hypothesis beats verbatim storage, is predicted almost exactly by a combinatorial identifiability bound on the selected operator's order and degree. It is invariant to term magnitude: scaling Fibonacci over twelve orders of magnitude leaves nd unchanged, because a hypothesis must encode its own initial conditions and the magnitude cancels. Second, at scale the learner exhibits a regime our curated corpus could not produce even once: across 20,000 OEIS sequences, 89.98% of those that fit a recurrence on some prefix fit none at full length. We call this the wilderness -- induction acquires a theory, loses it, and never recovers. Third, evaluating three language models on sequences stratified by these MDL regimes refuted our pre-registered hypothesis: models do not confabulate where MDL reports no theory, but hedge appropriately. Confident errors are inverted, concentrating on the easy stratum, where apparent competence tracks recognition of the sequence rather than induction of its rule. OEIS-derived benchmarks therefore substantially measure memorisation, and MDL supplies a cheap, contamination-free difficulty signal they currently lack. Code and data are released.

数学推理记忆机制基准测试描述长度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。