arXiv:2601.05280cs.ITcs.AI2026-01被引 2

大模型不是真正意义上的归纳推理器,自改进也难达奇点。

On Solomonoff Induction in Large Language Models and the Limits of Self-Improving: The Singularity Is Not Near Without Symbolic Model Synthesis

  • 用交叉熵等目标训练的模型无法实现索洛蒙诺夫归纳。
  • 算力提升不改变归纳原则,不能自动变优。
  • 未来需符号合成技术才能突破当前局限,适合研究者参考。

大语言模型是否能实现索洛蒙诺夫归纳,已成为算法信息论与机器学习交叉领域的核心问题。尽管自改进系统设想中存在正反馈机制,但现有基于交叉熵、负对数似然等目标的模型仅优化对给定条件分布的拟合,而非程序加权的通用混合分布。即使增加计算资源,也无法在不改变超参数或架构的前提下,使模型成为索洛蒙诺夫与莱文意义下的最优预测器。虽然数据处理不等式和莱文无增长定律依然成立,但对有限学习者和观察者而言,理论边界影响减弱,导致方法间出现偏差而非违反信息守恒。本文将不同资源受限估计器视为机制搜索的有限工具,其差异是发散而非违反。当前前沿模型向神经符号方向发展,强调模型合成,这可能为实现索洛蒙诺夫估计器提供路径。

原文摘要 · Abstract (English)

On the one hand, the question of whether large language models (LLMs) are Solomonoff induction estimators has become an explicit question at the intersection of Algorithmic Information Theory (AIT) and Machine Learning (ML) of great interest. On the other hand, the now old idea of an AI Singularity that requires a reliable positive-feedback process in which a system can generate, evaluate and retain genuine improvements to itself continues to come up and is a recurrent concept in the discussion of AGI. We connect and provide some answers to these issues based on current assumptions and future developments of neurosymbolic ML. We will demonstrate that cross-entropy, negative log-likelihood and cognate next-token objectives do not or cannot, by themselves, implement Solomonoff induction: they optimise fit to a supplied conditional distribution rather than a program-weighted universal mixture. While more compute within a fixed objective can improve fit without changing the inductive principle, additional computational resources do not intrinsically without external hyper-parameter or architectural changes, behave as optimal predictors in the Solomonoff and Levin sense. While the data-processing inequality (DPI) and Levin non-growth remain valid, we will show that for finite learners and finite observers, theoretical boundaries have less relevancy and generate a drift between possible approaches. To this end, we interpret different resource-bounded estimators as finite tools for mechanism search that show divergence, not violation, of (algorithmic) information conservation laws. A neurosymbolic direction taken by current frontier-model developers points towards the adoption of model synthesis and no longer purely statistical approaches to LLMs but where Solomonoff estimators are possible.

大模型索洛蒙诺夫符号合成归纳推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。