通过正则化汉克尔奇异值，让状态空间模型更易压缩且保持高精度。
Hankel Singular Value Regularization for Highly Compressible State Space Models
- 用汉克尔奇异值正则化，促使奇异值快速衰减，提升模型可压缩性
- 在长序列任务上压缩比达10倍，准确率不下降
- 针对块对角结构设计高效算法，支持训练时实时计算
使用状态空间模型作为层的深度神经网络适用于长序列任务，但训练后压缩困难。本文提出正则化状态空间模型的汉克尔奇异值之和，能促进奇异值快速衰减，从而实现高可压缩性。为使该正则化方法可扩展，我们利用状态空间模型参数化中系统矩阵的块对角结构，开发出一种高效计算汉克尔奇异值的算法，可在训练过程中快速完成。在Long Range Arena基准上的实验表明，经过正则化的状态空间层相比标准层最多可压缩10倍，同时保持高准确率。
原文摘要 · Abstract (English)
Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$\times$ more compressible than standard state space layers while maintaining high accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。