用KAN替换ViT中的MLP，缓解持续学习中的灾难性遗忘。
Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning
- 用基于样条的KAN替代ViT中的MLP，实现局部参数更新。
- 在MNIST和CIFAR100上显著提升旧任务准确率，保留知识能力更强。
- 适合研究持续学习、模型鲁棒性及ViT改进的学者参考。
持续学习（CL）指模型在不遗忘已有知识的前提下学习新任务，是人工智能中的关键挑战，尤其对使用多层感知机（MLPs）进行全局表征学习的视觉变压器（ViTs）而言。灾难性遗忘——新信息覆盖旧知识——在此类模型中尤为严重。本研究提出用科尔莫戈罗夫-阿诺德网络（KAN）替代ViTs中的MLP，以缓解该问题。KAN通过基于样条的激活函数实现局部可塑性，使每次样本仅更新部分参数，从而保护已学知识。研究在基准数据集（MNIST、CIFAR100）上评估了基于KAN的ViTs在持续学习场景下的表现，重点考察其在适应新任务的同时保持旧任务准确率的能力。实验结果表明，基于KAN的ViTs显著减轻了灾难性遗忘，在知识保留与任务适应方面均优于传统MLP-based ViTs。这一创新性集成为动态环境中的更鲁棒、自适应模型提供了可行路径。
原文摘要 · Abstract (English)
Continual learning (CL), the ability of a model to learn new tasks without forgetting previously acquired knowledge, remains a critical challenge in artificial intelligence, particularly for vision transformers (ViTs) utilizing Multilayer Perceptrons (MLPs) for global representation learning. Catastrophic forgetting, where new information overwrites prior knowledge, is especially problematic in these models. This research proposes replacing MLPs in ViTs with Kolmogorov-Arnold Network (KANs) to address this issue. KANs leverage local plasticity through spline-based activations, ensuring that only a subset of parameters is updated per sample, thereby preserving previously learned knowledge. The study investigates the efficacy of KAN-based ViTs in CL scenarios across benchmark datasets (MNIST, CIFAR100), focusing on their ability to retain accuracy on earlier tasks while adapting to new ones. Experimental results demonstrate that KAN-based ViTs significantly mitigate catastrophic forgetting, outperforming traditional MLP-based ViTs in knowledge retention and task adaptation. This novel integration of KANs into ViTs represents a promising step toward more robust and adaptable models for dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。