深度影响线性状态空间模型表达能力，深模型在参数约束下更优。
The Effect of Depth on the Expressivity of Deep Linear State-Space Models
- 通过构造法证明深模型可用更小参数范数模拟浅模型
- 参数量相当时，深度与宽度对表达力影响等价
- 理论给出最小深度上限，适合研究模型容量的学者
深度状态空间模型（SSMs）在序列建模中日益流行。尽管已有大量关于浅层SSMs的理论研究，但深度如何影响其表达能力仍是关键问题。本文系统研究了深度与宽度对深层线性SSMs的影响，旨在刻画其对模型表达能力的作用。首先，我们严格证明：在无参数约束时，增加深度与增加宽度在参数量同阶条件下通常等价。然而，在参数范数受约束的假设下，深度与宽度的影响显著不同。我们展示了具有大参数范数的浅层线性SSM可被一个参数范数更小的深层线性SSM通过构造方法表示。特别地，这表明在范数约束下，深层SSMs比浅层模型更能表示高范数目标。最后，我们推导出在参数范数约束下，表示给定浅层线性SSM所需的最小深度上界，并通过数值实验验证了理论结果。
原文摘要 · Abstract (English)
Deep state-space models (SSMs) have gained increasing popularity in sequence modelling. While there are numerous theoretical investigations of shallow SSMs, how the depth of the SSM affects its expressiveness remains a crucial problem. In this paper, we systematically investigate the role of depth and width in deep linear SSMs, aiming to characterize how they influence the expressive capacity of the architecture. First, we rigorously prove that in the absence of parameter constraints, increasing depth and increasing width are generally equivalent, provided that the parameter count remains within the same order of magnitude. However, under the assumption that the parameter norms are constrained, the effects of depth and width differ significantly. We show that a shallow linear SSM with large parameter norms can be represented by a deep linear SSM with smaller norms using a constructive method. In particular, this demonstrates that deep SSMs are more capable of representing targets with large norms than shallow SSMs under norm constraints. Finally, we derive upper bounds on the minimal depth required for a deep linear SSM to represent a given shallow linear SSM under constrained parameter norms. We also validate our theoretical results with numerical experiments
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。