用可学习的边函数替代固定激活,提升小语言模型的可解释性。
Kolmogorov--Arnold Networks for Small Language Models
- 用B样条KAN重构88万条前馈边,87.8%活性高于0.1
- 剪枝20%-25%低活性边几乎不损失性能
- 适合需要审计模型内部变换的场景
Kolmogorov-Arnold网络(KANs)将固定节点激活替换为可学习的一维边函数,提供显式的可解释接口,并可能替代Transformer的前馈网络。在六层、1000万参数的B样条KAN中,我们重构了全部884,736条前馈边:87.8%的边活性大于0.1,0.4%处于不活跃状态。剪掉活性最低的20%-25%边仅导致微小损失增加,尽管结构化MLP神经元剪枝也能承受类似稀疏度。该审计在BabyLM上复现,但网格大小扫描表明,近似全压缩和高闭式拟合覆盖率是低容量网格-2基底的特性,非普遍的KAN行为。作为替代,我们在BabyLM上评估了MLP、SwiGLU、分组切比雪夫及有门控的GR-KAN网络。KAN族与门控变体在验证损失上优于GELU MLP,但此排序未推广至标准基准:在十次种子和59,875个BLiMP对上,准确率范围为62.4%-63.1%,EWoK仍处于随机水平,(+0.7)点的GR-KAN优势在附录中反转。更大规模测试亦具警示意义:参数匹配的MLPEdge在Wikitext-103上表现不如原MLP,2.86亿参数的GR-KAN在稳定后仍低于SwiGLU ClimbMix基线。因此,小基底KAN提供了实用且可跨语料库迁移的可解释接口,但所测替代方案在基准测试、质量或延迟上均无一致优势。
原文摘要 · Abstract (English)
Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8\% exceed (NLS>0.1) and 0.4\% are inactive. Pruning the lowest-activity 20--25\% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1\%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。