用B样条解耦压缩Transformer模型,参数量大幅减少且精度保持稳定。
Robust Basis Spline Decoupling for the Compression of Transformer Models

- 基于B样条构建解耦框架,提升非线性函数表达能力与数值稳定性。
- 在Vision和Swin Transformer上实现显著参数压缩,精度损失可控。
- 适合需要轻量化部署的Transformer模型优化场景。
解耦是一种将多元函数表示为线性变换与一元非线性函数复合的强大建模范式。单层解耦可视为带灵活激活函数的单隐藏层全连接网络,与神经网络直接关联。因此,解耦方法在神经网络压缩领域日益受到关注,因其能实现结构化近似并降低参数复杂度。现有基于张量的解耦方法通常依赖多项式或分段线性参数化内部非线性函数,易引发数值不稳或表达力不足。本文提出一种基于B样条的解耦框架,通过利用B样条的局部支撑性和灵活光滑控制,实现更稳定、更具表现力的表示。我们推导出一种约束耦合矩阵-张量分解(R-CMTF-BSD),并提出一种鲁棒交替最小二乘算法,结合归一化与Tikhonov正则化。实验在合成数据及Transformer模型压缩上验证了该方法的有效性。在Vision和Swin Transformer上的结果表明,基于B样条的解耦能实现显著参数缩减,同时保持竞争性精度,使R-CMTF-BSD成为结构化神经网络压缩的有力工具。
原文摘要 · Abstract (English)
Decoupling is a powerful modeling paradigm for representing multivariate functions as compositions of linear transformations and univariate nonlinear functions. A single-layer decoupling can be viewed as a fully connected neural network with a single hidden layer and flexible activation functions, providing a direct link with neural networks. Because of this, the use of decoupling methods has gained increasing attention in neural network domains, particularly compression, since it enables structured approximations with reduced parameter complexity. Existing tensor-based decoupling methods typically rely on polynomial or piecewise-linear parameterizations of the internal nonlinear functions, which can suffer from numerical instability or limited expressiveness. In this work, we introduce a B-spline-based decoupling framework that generalizes these existing approaches. By exploiting the local support and flexible smoothness control of B-splines, the proposed formulation yields a more numerically stable and expressive representation. We derive a constrained coupled matrix-tensor factorization and propose a robust alternating least-squares algorithm, called R-CMTF-BSD, incorporating normalization and Tikhonov regularization. The proposed method is validated through experiments on synthetic data and transformer model compression. Results on the Vision and Swin Transformer architectures demonstrate that B-spline decoupling enables substantial parameter reduction while maintaining competitive accuracy, making the R-CMTF-BSD algorithm a promising tool for structured neural network compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。