将大模型视为马尔可夫链，揭示其推理机制与错误根源
Large Language Models as Markov Chains
- 把自回归语言模型看作有限状态的马尔可夫链，从原理出发分析多步推理
- 理论推导出预训练和上下文学习的泛化界，与Llama/Gemma实测行为吻合
- 解释高温生成导致重复、混乱等异常现象，适合研究模型机理者阅读
大型语言模型（LLMs）在众多自然语言处理任务中表现出色，甚至超越语言范畴。然而，其泛化能力的系统性理论分析仍不完善。本文通过建立基于自回归Transformer的语言模型与有限状态空间马尔可夫链之间的等价关系，从第一性原理出发研究了LLM的多步推理机制。我们据此解释了高温度生成时出现的重复、不连贯等异常行为。进一步，在合理数据与模型假设下，推导出预训练与上下文学习的泛化边界。在最新Llama与Gemma系列模型上的实验表明，该理论能准确捕捉实际行为。
原文摘要 · Abstract (English)
Large language models (LLMs) are remarkably efficient across a wide range of natural language processing tasks and well beyond them. However, a comprehensive theoretical analysis of the LLMs' generalization capabilities remains elusive. In our paper, we approach this task by drawing an equivalence between autoregressive transformer-based language models and Markov chains defined on a finite state space. This allows us to study the multi-step inference mechanism of LLMs from first principles. We relate the obtained results to the pathological behavior observed with LLMs such as repetitions and incoherent replies with high temperature. Finally, we leverage the proposed formalization to derive pre-training and in-context learning generalization bounds for LLMs under realistic data and model assumptions. Experiments with the most recent Llama and Gemma herds of models show that our theory correctly captures their behavior in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。