用多种激活函数组合提升KAN的表示能力，显著优于MLP和传统KAN。
AF-KAN: Activation Function-Based Kolmogorov-Arnold Networks for Efficient Representation Learning
- 基于激活函数组合构建新KAN，扩展ReLU-KAN处理多输入能力
- 在图像分类上以更少参数实现超越MLP和传统KAN的性能
- 适合追求高效表示学习的模型设计者，尤其关注结构创新
Kolmogorov-Arnold Networks (KANs) 激发了众多科学问题中的应用探索，有潜力替代多层感知机（MLPs）。尽管许多KAN使用基函数或多项式函数（如B样条），但ReLU-KAN通过组合ReLU函数模拟B样条结构并利用其计算速度。然而，ReLU-KAN不适用于多输入，且因ReLU对负值处理受限，影响特征提取。为此，本文提出基于激活函数的Kolmogorov-Arnold网络（AF-KAN），在ReLU-KAN基础上引入多种激活函数及其组合，并结合注意力机制与数据归一化实现参数压缩。通过测试不同激活函数、函数组合、网格大小和样条阶数，验证了AF-KAN的有效性并确定最优配置。实验表明，AF-KAN在相同参数量下显著优于MLP、ReLU-KAN及其他KAN；即使参数量仅为6至10倍时仍具竞争力。但训练时间更长，消耗更多浮点运算量。代码已开源：https://github.com/hoangthangta/All-KAN。
原文摘要 · Abstract (English)
Kolmogorov-Arnold Networks (KANs) have inspired numerous works exploring their applications across a wide range of scientific problems, with the potential to replace Multilayer Perceptrons (MLPs). While many KANs are designed using basis and polynomial functions, such as B-splines, ReLU-KAN utilizes a combination of ReLU functions to mimic the structure of B-splines and take advantage of ReLU's speed. However, ReLU-KAN is not built for multiple inputs, and its limitations stem from ReLU's handling of negative values, which can restrict feature extraction. To address these issues, we introduce Activation Function-Based Kolmogorov-Arnold Networks (AF-KAN), expanding ReLU-KAN with various activations and their function combinations. This novel KAN also incorporates parameter reduction methods, primarily attention mechanisms and data normalization, to enhance performance on image classification datasets. We explore different activation functions, function combinations, grid sizes, and spline orders to validate the effectiveness of AF-KAN and determine its optimal configuration. In the experiments, AF-KAN significantly outperforms MLP, ReLU-KAN, and other KANs with the same parameter count. It also remains competitive even when using fewer than 6 to 10 times the parameters while maintaining the same network structure. However, AF-KAN requires a longer training time and consumes more FLOPs. The repository for this work is available at https://github.com/hoangthangta/All-KAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。