Mamba和状态空间模型的计算能力被证明与Transformer相当，无法解决某些复杂问题。
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
- 通过电路复杂度分析，证明Mamba和SSM属于TC⁰类计算能力
- 在多项式精度和常数深度下，无法解决算术公式等难题
- 挑战了Mamba比Transformer更强大表达能力的主流假设
本文基于电路复杂度框架，分析了Mamba和状态空间模型（SSMs）的计算局限性。尽管Mamba具有状态记忆设计且近期被视为超越Transformer的有力候选，我们证明：当使用$ℍ{poly}(n)$-精度且层深为常数时，两者均位于$ℍ{DLOGTIME}$-uniform $ℍ{TC}^0$复杂度类中。这一结果表明，从理论上看Mamba的计算能力与Transformer一致，若$ℍ{TC}^0 eq ℍ{NC}^1$，则无法求解算术公式问题、布尔公式值问题及置换组合问题。研究贡献包括严谨证明：选择性SSM与Mamba架构可被$ℍ{DLOGTIME}$-uniform $ℍ{TC}^0$电路模拟，且无法处理$ℍ{TC}^0$之外的问题。
原文摘要 · Abstract (English)
In this paper, we analyze the computational limitations of Mamba and State-space Models (SSMs) by using the circuit complexity framework. Despite Mamba's stateful design and recent attention as a strong candidate to outperform Transformers, we have demonstrated that both Mamba and SSMs with $\mathrm{poly}(n)$-precision and constant-depth layers reside within the $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$ complexity class. This result indicates Mamba has the same computational capabilities as Transformer theoretically, and it cannot solve problems like arithmetic formula problems, boolean formula value problems, and permutation composition problems if $\mathsf{TC}^0 \neq \mathsf{NC}^1$. Therefore, it challenges the assumption Mamba is more computationally expressive than Transformers. Our contributions include rigorous proofs showing that Selective SSM and Mamba architectures can be simulated by $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$ circuits, and they cannot solve problems outside $\mathsf{TC}^0$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。