用控制理论方法压缩状态空间模型，参数减到1/32仍保持性能。
Compression Method for Deep Diagonal State Space Model Based on $H^2$ Optimal Reduction
- 将控制理论中的H²约化技术用于压缩线性状态空间模块。
- 在LRA基准上参数减少至1/32，性能不降。
- 适合部署在资源受限设备上的长序列建模任务。
结合线性状态空间模型（SSM）的深度学习模型因能捕捉序列数据中的长程依赖而受到关注，但其庞大的参数量给资源受限设备的部署带来挑战。本文提出一种高效参数压缩方法，将控制理论中的H²最优约化技术应用于线性SSM组件。实验表明，在LRA基准上，该方法相比现有基于平衡截断（Balanced Truncation）的方法表现更优，同时将SSM参数量压缩至原始模型的1/32，且未牺牲原模型性能。
原文摘要 · Abstract (English)
Deep learning models incorporating linear SSMs have gained attention for capturing long-range dependencies in sequential data. However, their large parameter sizes pose challenges for deployment on resource-constrained devices. In this study, we propose an efficient parameter reduction method for these models by applying $H^{2}$ model order reduction techniques from control theory to their linear SSM components. In experiments, the LRA benchmark results show that the model compression based on our proposed method outperforms an existing method using the Balanced Truncation, while successfully reducing the number of parameters in the SSMs to $1/32$ without sacrificing the performance of the original models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。