arXiv:2608.01005cs.LGcs.AI2026-08

提出层次化索洛蒙归纳,让模型能从数据中学习并最优预测序列。

Hierarchical Solomonoff Induction: An Unbounded Machine Learning Model

  • 用超先验构建可条件化的层次模型,融合所有索洛蒙先验
  • 证明其误差随数据量增长趋于0,极限下预测最优
  • 适合研究理想机器学习模型的理论学者

索洛蒙归纳(SolInd)提供了理想的无界序列预测模型,但难以自然描述大型语言模型基于训练数据的外推行为。本文通过德·芬内蒂定理对交换分布的应用,将SolInd扩展为层次化索洛蒙归纳(HSI),在所有索洛蒙先验上建立超先验,并可对已观察序列进行条件化。我们扩展了Wood等人的证明,表明这些混合的通用混合也等价于SolInd,从而证明HSI=SolInd。此外,我们证明了HSI在任意分布上的额外误差,由其真实生成器在超先验中的复杂度所界定。该结果与SolInd的预测误差被序列的柯尔莫哥洛夫复杂度限制直接可比,且保证随着数据集增大,平均额外误差趋于0,实现极限下的最优预测。我们主张HSI是给定数据集时序列预测的理想无界模型,正如SolInd是单个序列的理想模型。

原文摘要 · Abstract (English)

Solomonoff Induction, or SolInd, provides an ideal unbounded model of a priori sequence prediction but cannot naturally describe extrapolation from a given training dataset, as performed by Large Language Models. We apply de Finetti's theorem on exchangeable distributions to SolInd to produce what we call Hierarchical Solomonoff Induction, or HSI, which maintains a hyperprior over all Solomonoff priors that can be conditioned on previously observed sequences. We extend Wood et al.'s proof that universal mixtures of semimeasures are equivalent to SolInd to show that universal mixtures of these mixtures are also equivalent, proving that HSI=SolInd. We also prove that HSI's excess error on any distribution, compared to its true generator, is bounded by that generator's complexity in the hyperprior. This result is directly comparable to SolInd's prediction error being bounded by the Kolmogorov complexity of the sequence being predicted, and forces HSI's average excess error to converge to 0 as a dataset grows, leading to optimal prediction in the limit. We claim that HSI is an ideal unbounded model of sequence prediction given a dataset in the same way that SolInd is ideal over individual sequences.

序列预测理论模型先验建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。