揭示多层状态空间模型在组合任务中的表达局限与突破路径
On the Expressive Power and Limitations of Multi-Layer SSMs
- 发现多层SSM在组合任务中存在根本性表达瓶颈
- 在线思维链可使其表达能力媲美流式算法
- 宽度与精度不可互换，但引入在线思维链后可等价
我们研究了多层状态空间模型（SSMs）的表达能力与局限。首先，发现多层SSM在组合任务中存在根本性限制，暴露出其与流式模型之间的本质差距。其次，考察了思维链（CoT）的作用，表明离线思维链无法根本提升表达能力，而在线思维链则能显著增强其能力。事实上，引入在线思维链后，多层SSM的表达能力等价于流式算法。最后，研究了宽度与精度之间的权衡，发现在基础模型中二者不可互换，但一旦允许在线思维链，便可实现清晰的等价关系。总体而言，我们的结果为深度、有限精度和思维链如何共同塑造SSM的表达力与边界提供了统一视角。
原文摘要 · Abstract (English)
We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。