用加法多步马尔可夫链缓解大模型的高维状态困境
Additive Multi-Step Markov Chains and the Curse of Dimensionality in Large Language Models
- 将多步依赖拆解为多个历史深度的叠加贡献，避免组合爆炸
- 证明了加法型与带步进记忆函数的马尔可夫链等价
- 首次在加法模型中引入信息温度概念，适用于大模型分析
大规模语言模型（LLMs）处于极高维的状态空间中，词元嵌入及其隐藏表示形成了复杂依赖关系，难以简化为经典马尔可夫结构。本文探索了使用N阶加法马尔可夫链对LLM动态进行理论可行的近似。该模型将下一个词元的条件概率分解为多个历史深度贡献的叠加，有效缓解了高阶马尔可夫过程典型的组合爆炸问题。主要成果是建立了加法多步链与具有步进记忆函数的链之间的对应关系，由此引入了信息温度概念，不仅适用于步进型，也适用于加法型N阶马尔可夫链。
原文摘要 · Abstract (English)
Large-scale language models (LLMs) operate in extremely high-dimensional state spaces, where both token embeddings and their hidden representations create complex dependencies that are not easily reduced to classical Markov structures. In this paper, we explore a theoretically feasible approximation of LLM dynamics using N-order additive Markov chains. Such models allow the conditional probability of the next token to be decomposed into a superposition of contributions from multiple historical depths, reducing the combinatorial explosion typically associated with high-order Markov processes. The main result of the work is the establishment of a correspondence between an additive multi-step chain and a chain with a step-wise memory function. This equivalence allowed the introduction of the concept of information temperature not only for stepwise but also for additive N-order Markov chains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。