线性动态系统学习存在参数临界值，低于则无法稳定预测长序列。
State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?
- 通过线性动态系统模拟自监督学习，发现参数量达临界值才可稳定建模。
- 长程相关任务需至少特定数量参数，否则误差随序列变长而发散。
- 适用于研究模型规模与能力涌现关系的理论工作者。
大语言模型在参数量达到临界规模时会涌现出多步推理等能力。本文探讨这一现象是否能在更简单的理论模型中复现。研究发现，学习线性动态系统——一种自监督学习的简化形式——也存在相变现象：对于非遍历性系统，当学习者参数少于某一临界阈值时，无法在长序列长度下实现有界误差。换言之，具有显著长程相关性的任务需要至少特定数量的参数，这类似于能力涌现。此外，研究了学习器参数化的影响，考虑一个带有隐状态的简单线性系统（即ℝ上的不完全观测随机游走）。结果表明，若线性滤波器长度未超过依赖有效记忆长度和问题时域的阈值，则无法成功学习该随机游走。
原文摘要 · Abstract (English)
How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems -- a simple instance of self-supervised learning -- exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters -- a phenomenon akin to emergence. We also investigate the role of the learner's parametrization and consider a simple version of a linear dynamical system with hidden state -- an imperfectly observed random walk in $\mathbb{R}$. For this situation, we show that there exists no learner using a linear filter which can succesfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。