通过压缩基函数、神经元和精度,提升KAN网络的效率与速度。
SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits

- 用可学习门控机制在三个维度上同时压缩KAN网络。
- 8位量化普遍有效,物理压缩可减少73%参数且不降性能。
- 适合需要高效推理的边缘设备或硬件部署场景。
Kolmogorov--Arnold Networks (KANs) 将标量权重替换为可学习的一维函数,由多个基系数参数化,引入了传统神经网络压缩无法直接处理的冗余。本文提出 extbf{SparseKAN},一种统一方法,从基函数、神经元/通道、数值精度三个互补维度压缩 KAN。SparseKAN 为基分支、非线性基分支及单个基项配备分层可学习门控,并在可微分激活代价目标下训练。学习到的重要结构随后在显式基函数和宽度预算下固化,恢复为全精度或低精度形式,并物理紧凑为更小的稠密张量而非保留稀疏掩码。在 MNIST、CIFAR-10、CIFAR-100 上对样条、多项式、RBF、小波、卷积 KAN 变体的实验表明,各结构性轴的成本可预测地组合。还发现基函数依赖的项重要性差异:基于系数的选择相比等阶数截断最高提升15.25准确率点(多项式设定)。8位量化广泛鲁棒,而4位卷积KAN需量化感知适配。物理压缩在MNIST上移除高达73.0%参数且无精度损失,大批次CUDA延迟降至稠密执行的0.51倍。在ZCU104 FPGA上,最终稀疏低比特模型实现最高23.63倍更低推理延迟,证明 SparseKAN 成功将函数冗余转化为可观的软硬件效率。代码开源于 https://github.com/OSU-STARLAB/SparseKAN。
原文摘要 · Abstract (English)
Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions parameterized by multiple basis coefficients. This introduces a source of redundancy that conventional neural-network compression does not directly expose. We present \textbf{SparseKAN}, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision. SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective. The learned importance structure is subsequently hardened under explicit basis and width budgets, recovered in full or low precision, and physically compacted into smaller dense tensors rather than retained as sparse masks. Experiments on MNIST, CIFAR-10, and CIFAR-100 across spline, polynomial, RBF, wavelet, and convolutional KAN variants show that the structural axes compose predictably in cost. We also find strong basis-dependent differences in term importance: coefficient-based selection outperforms matched low-order truncation by up to 15.25 accuracy points in the evaluated Gram-polynomial settings. Eight-bit quantization is broadly robust, whereas 4-bit convolutional KANs require quantization-aware adaptation. Physical compaction removes up to 73.0\% of parameters without accuracy loss on MNIST and reduces large-batch CUDA latency to as little as $0.51\times$ dense execution. On a ZCU104 FPGA, the resulting sparse low-bit models achieve up to $23.63\times$ lower inference latency, demonstrating that SparseKAN converts functional redundancy into measurable software and hardware efficiency. The SparseKAN implementation is available at https://github.com/OSU-STARLAB/SparseKAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。