Mamba能高效学习马尔可夫链的最优平滑估计，揭示其理论优势。
From Markov to Laplace: How Mamba In-Context Learns Markov Chains
- 单层Mamba通过卷积实现最优拉普拉斯平滑估计
- 在上下文学习中表现接近贝叶斯最优，性能超越传统方法
- 适合研究序列建模与模型可解释性的研究人员
尽管基于Transformer的语言模型推动了人工智能的发展，但其计算复杂性促使人们关注替代方案，如结构化状态空间序列模型（SSMs）和选择性SSMs。其中，Mamba（S6）及其变体Mamba-2在推理速度上相比Transformer有显著提升，同时在复杂语言建模任务中达到或超越了其性能。然而，尽管架构创新和实证成功，Mamba的基本学习能力仍不清晰。本文通过研究在上下文学习（ICL）中对马尔可夫链的学习，发现即使单层Mamba也能高效学习上下文中的拉普拉斯平滑估计器，该估计器既是贝叶斯最优也是极小极大最优。我们从理论上刻画了Mamba的表征能力，揭示卷积在实现最优拉普拉斯平滑中的根本作用。这些理论洞察与实证结果高度一致，据我们所知，这是首次将Mamba与最优统计估计器建立正式联系。最后，我们提出了受此发现启发的未来研究方向。
原文摘要 · Abstract (English)
While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structured state space sequence models (SSMs) and Selective SSMs. Among these, Mamba (S6) and its variant Mamba-2 have shown remarkable inference speed-ups over transformers while achieving comparable or superior performance on complex language modeling tasks. However, despite these architectural innovations and empirical successes, the fundamental learning capabilities of Mamba remain poorly understood. In this paper, we address this gap by studying in-context learning (ICL) on Markov chains and uncovering an interesting phenomenon: even a single-layer Mamba efficiently learns the in-context Laplacian smoothing estimator, which is both Bayes and minimax optimal. To explain this, we theoretically characterize the representation capacity of Mamba and reveal the fundamental role of convolution in enabling it to represent the optimal Laplacian smoothing. These theoretical insights align strongly with empirical results and, to the best of our knowledge, represent the first formal connection between Mamba and optimal statistical estimators. Finally, we outline promising research directions inspired by these findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。