arXiv:2503.01124cs.CV2025-03被引 1

用柯尔莫哥洛夫网络替换ViT的MLP,提升非线性建模能力

ViKANformer: Embedding Kolmogorov Arnold Networks in Vision Transformers for Pattern-Based Learning

  • 将ViT中MLP层替换为多种KAN网络结构
  • 在MNIST上实现超97%准确率,但训练开销增加
  • 适合追求高精度且不敏感计算成本的研究者

视觉变换器(ViT)通过在图像块嵌入上应用自注意力机制,在图像分类任务中取得显著进展。然而,每个变换器层中的标准MLP子层可能无法最优捕捉复杂的非线性依赖关系。本文提出ViKANformer,一种将MLP子层替换为柯尔莫哥洛夫-阿诺德网络(KAN)扩展的视觉变换器,包括Vanilla KAN、Efficient-KAN、Fast-KAN、SineKAN和FourierKAN,并研究了Flash Attention变体。基于柯尔莫哥洛夫-阿诺德定理,该定理保证多元连续函数可表示为一元连续函数之和,旨在增强模型表达能力。在MNIST数据集上的实验表明,SineKAN、Fast-KAN及调优后的Vanilla KAN均能实现超过97%的准确率,尽管训练开销有所增加。这一权衡表明,当计算成本可接受时,KAN扩展具有潜在优势。论文详细描述了各类扩展结构,给出了训练/测试准确率以及F1/ROC指标,并提供伪代码与超参数以支持复现。最后,将ViKANformer与简单MLP和小型CNN基线在MNIST上对比,展示了即使在小规模数据集上,基于Transformer的方法仍具高效性。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) have significantly advanced image classification by applying self-attention on patch embeddings. However, the standard MLP blocks in each Transformer layer may not capture complex nonlinear dependencies optimally. In this paper, we propose ViKANformer, a Vision Transformer where we replace the MLP sub-layers with Kolmogorov-Arnold Network (KAN) expansions, including Vanilla KAN, Efficient-KAN, Fast-KAN, SineKAN, and FourierKAN, while also examining a Flash Attention variant. By leveraging the Kolmogorov-Arnold theorem, which guarantees that multivariate continuous functions can be expressed via sums of univariate continuous functions, we aim to boost representational power. Experimental results on MNIST demonstrate that SineKAN, Fast-KAN, and a well-tuned Vanilla KAN can achieve over 97% accuracy, albeit with increased training overhead. This trade-off highlights that KAN expansions may be beneficial if computational cost is acceptable. We detail the expansions, present training/test accuracy and F1/ROC metrics, and provide pseudocode and hyperparameters for reproducibility. Finally, we compare ViKANformer to a simple MLP and a small CNN baseline on MNIST, illustrating the efficiency of Transformer-based methods even on a small-scale dataset.

视觉TransformerKAN网络非线性建模图像分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。