arXiv:2504.15758cs.LGcs.SY2025-04

提出可观察性约束新方法,提升神经状态空间模型训练效率与稳定性。

Observability conditions for neural state-space models with eigenvalues and their roots of unity

  • 基于控制理论设计可学习矩阵的可观测性约束策略
  • 利用傅里叶变换与范德蒙德矩阵实现高概率可观测性
  • 适用于Mamba等架构,支持大规模高效训练

本文从常微分方程与控制理论视角,研究神经状态空间模型及Mamba架构中的可观测性问题。提出针对可学习初始隐状态、连续时间与高维场景的可观测性强化方法,强调特征值与单位根的作用。方法在计算上高效,可扩展至大规模系统。基于经典控制理论构建机器学习中的可观测性条件,并分析其计算复杂度。主要成果包括:通过置换实现无需高精度矩阵的可观测性;利用傅里叶变换与特征结构,在随机学习下以高概率保证可观测性;为Mamba提出类Hautus条件,但采用范德蒙德矩阵替代特征向量;设计共享参数构造,提升高次幂计算效率;开发满足Robbins-Monro条件的训练算法,在特定正交性下优于传统方法。

原文摘要 · Abstract (English)

We operate through the lens of ordinary differential equations and control theory to study the concept of observability in the context of neural state-space models and the Mamba architecture. We develop strategies to enforce observability, which are tailored to a learning context, specifically where the hidden states are learnable at initial time, in conjunction to over its continuum, and high-dimensional. We also highlight our methods emphasize eigenvalues, roots of unity, or both. Our methods effectuate computational efficiency when enforcing observability, sometimes at great scale. We formulate observability conditions in machine learning based on classical control theory and discuss their computational complexity. Our nontrivial results are fivefold. We discuss observability through the use of permutations in neural applications with learnable matrices without high precision. We present two results built upon the Fourier transform that effect observability with high probability up to the randomness in the learning. These results are worked with the interplay of representations in Fourier space and their eigenstructure, nonlinear mappings, and the observability matrix. We present a result for Mamba that is similar to a Hautus-type condition, but instead employs an argument using a Vandermonde matrix instead of eigenvectors. Our final result is a shared-parameter construction of the Mamba system, which is computationally efficient in high exponentiation. We develop a training algorithm with this coupling, showing it satisfies a Robbins-Monro condition under certain orthogonality, while a more classical training procedure fails to satisfy a contraction with high Lipschitz constant.

状态空间可观测性Mamba控制理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。