大模型能从上下文推断马尔可夫链转移概率，而非死记硬背。
Markov Chain Estimation with In-Context Learning
- 用上下文学习让transformer自适应不同转移矩阵
- 模型规模和训练数据量超过阈值后可泛化预测
- 状态编码越复杂，对新结构的鲁棒性越强
我们研究了transformer在仅通过下一个词预测训练时，学习涉及上下文算法的能力。构建了随机转移矩阵的马尔可夫链，在训练和测试时使用不同矩阵。结果显示，当模型规模和训练集大小超过某一阈值时，模型能够从上下文中估计转移概率，而非记忆训练模式。此外，更复杂的状态编码可提升模型对训练中未见结构的马尔可夫链的预测鲁棒性。
原文摘要 · Abstract (English)
We investigate the capacity of transformers to learn algorithms involving their context while solely being trained using next token prediction. We set up Markov chains with random transition matrices and we train transformers to predict the next token. Matrices used during training and test are different and we show that there is a threshold in transformer size and in training set size above which the model is able to learn to estimate the transition probabilities from its context instead of memorizing the training patterns. Additionally, we show that more involved encoding of the states enables more robust prediction for Markov chains with structures different than those seen during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。