解析Mamba核心结构的泛化能力，揭示状态矩阵稳定性对模型性能的影响。
Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention
- 基于覆盖数推导选择性状态空间模型的泛化界。
- 发现连续时间状态矩阵的谱辐值影响训练稳定性和序列长度泛化能力。
- 在合成任务、情感分类和列表操作任务上验证理论预测的有效性。
状态空间模型（SSMs）近年来成为序列建模任务中对Transformer的有力替代。本文对Mamba模型的核心组件——选择性状态空间模型进行了理论泛化分析。基于近期关于Transformer的理论进展，我们推导出一种新的基于覆盖数的泛化边界。利用该结果，分析了连续时间状态矩阵的谱辐值如何影响模型在训练中的稳定性及其跨序列长度的泛化能力。我们在合成多数任务、IMDb情感分类基准和ListOps任务上对理论发现进行了实证验证，展示了理论洞察如何转化为实际模型行为。
原文摘要 · Abstract (English)
State-space models (SSMs) have recently emerged as a compelling alternative to Transformers for sequence modeling tasks. This paper presents a theoretical generalization analysis of selective SSMs, the core architectural component behind the Mamba model. We derive a novel covering number-based generalization bound for selective SSMs, building upon recent theoretical advances in the analysis of Transformer models. Using this result, we analyze how the spectral abscissa of the continuous-time state matrix influences the model's stability during training and its ability to generalize across sequence lengths. We empirically validate our findings on a synthetic majority task, the IMDb sentiment classification benchmark, and the ListOps task, demonstrating how our theoretical insights translate into practical model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。