用自身隐状态预测提升数据效率,理论证明只需常数样本即可学好深层结构。
Learn from your own latents and not from tokens: A sample-complexity theory
- 通过预测自身隐状态而非原始标记,实现更高效学习
- 在深度为L的树结构数据上,仅需常数级样本即可恢复结构
- 适合关注数据效率与模型可解释性的研究者
生成模型如扩散模型和大语言模型虽性能卓越,但训练所需数据量远超生物学习者。一种新范式是让网络预测自身对相关视图或掩码区域的隐状态表示(如data2vec和JEPA),该思想与皮层预测编码理论相关。尽管实证效果良好,其理论理解仍有限。核心问题包括:隐状态预测能否显著提升数据效率?多尺度层级堆叠是否有益?本文采用可解析的概率上下文无关文法作为数据,该文法通过深度为L的隐藏符号树递归生成可见标记序列。我们证明:监督学习或标记级自监督学习需要指数级于L的样本才能恢复隐树;而隐状态预测仅需常数级样本(含对数因子)。实验验证了该边界:(i) 层级聚类算法,(ii) 端到端神经网络,其预测-聚类模块在每层通过梯度下降预测自身隐状态,(iii) 首次对data2vec进行样本复杂度分析,发现其隐含执行层级隐状态预测。这表明显式堆叠如H-JEPA基本冗余。
原文摘要 · Abstract (English)
Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their \emph{own} latent representations of related views or masked regions, as in data2vec and JEPA -- an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth $L$. For such data, supervised or token-level SSL require a number of samples \emph{exponential} in $L$ to recover the latent tree; we prove that latent prediction achieves this with a number of samples \emph{constant} in $L$, up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。