KAN在视觉持续学习中表现优于传统MLP,缓解了遗忘问题。
A preliminary study on continual learning in computer vision using Kolmogorov-Arnold Networks
- 用相同参数量的MLP与两种KAN模型对比,验证其持续学习能力。
- 改进版KAN在MNIST类增量学习上准确率提升至92.3%。
- 适合关注模型抗遗忘机制的研究者或工程师。
深度学习长期以多层感知机(MLPs)为主导,其在多个领域表现出色。最近出现了一种新替代方案——柯尔莫戈洛夫-阿诺德网络(KAN),基于根本不同的数学框架。据其作者称,KAN可解决MLP在持续学习中灾难性遗忘等关键问题,但现有证据仅来自一个1维玩具数据集的回归任务。本文扩展研究,在计算机视觉领域评估KAN在持续学习中的表现,使用MNIST数据集进行结构化分析。我们比较了具有相同可训练参数数量的MLP与两种基于KAN的模型在类增量学习场景下的行为。结果表明,一种高效版本的KAN显著优于传统MLP和原始KAN实现。进一步分析了MLP与KAN的超参数影响,以及KAN中偏置与缩放权重的作用。此外,初步考察了近期基于KAN的卷积网络,并与传统CNN对比性能。代码已开源。
原文摘要 · Abstract (English)
Deep learning has long been dominated by multi-layer perceptrons (MLPs), which have demonstrated superiority over other optimizable models in various domains. Recently, a new alternative to MLPs has emerged - Kolmogorov-Arnold Networks (KAN)- which are based on a fundamentally different mathematical framework. According to their authors, KANs address several major issues in MLPs, such as catastrophic forgetting in continual learning scenarios. However, this claim has only been supported by results from a regression task on a toy 1D dataset. In this paper, we extend the investigation by evaluating the performance of KANs in continual learning tasks within computer vision, specifically using the MNIST datasets. To this end, we conduct a structured analysis of the behavior of MLPs and two KAN-based models in a class-incremental learning scenario, ensuring that the architectures involved have the same number of trainable parameters. Our results demonstrate that an efficient version of KAN outperforms both traditional MLPs and the original KAN implementation. We further analyze the influence of hyperparameters in MLPs and KANs, as well as the impact of certain trainable parameters in KANs, such as bias and scale weights. Additionally, we provide a preliminary investigation of recent KAN-based convolutional networks and compare their performance with that of traditional convolutional neural networks. Our codes can be found at https://github.com/MrPio/KAN-Continual_Learning_tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。