arXiv:2409.10463cs.LGstat.CO2024-09ICML被引 10

在数据少时,改良版MLP比KAN更准,激活函数设计是关键。

Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons

  • 给每个神经元设独特可学习激活函数,提升MLP表现
  • 样本仅100时,MLP准确率达0.91,远超KAN的0.53
  • 适合小数据场景下的模型选型与优化参考

多层感知机(MLPs)长期是深度学习的核心,能建模复杂关系。近年来,科尔莫戈罗夫-阿诺德网络(KANs)作为替代出现,其通过在边上的可学习激活函数实现高度灵活性,不同于传统以神经元为中心的设计。然而,KANs显著增加可学习参数量,引发其在数据稀缺环境下的有效性担忧。本文从算法与实验双角度对MLPs与KANs进行系统比较,重点关注低数据场景。我们提出一种有效方法,为每个神经元设计独立的参数化激活函数,使与KANs的对比更公平。基于模拟数据及医学与工程领域的两个真实数据集的实证评估显示,模型复杂度与精度间存在权衡,尤其受网络深度影响。结果表明,在样本量约一百时,采用个性化激活函数的MLPs在预测精度上显著优于默认超参数下的KANs——例如,在增材制造的三分类问题中,MLPs中位准确率达0.91,而KANs仅为0.53。这些发现揭示了激活函数选择对神经网络性能的关键作用。

原文摘要 · Abstract (English)

Multilayer Perceptrons (MLPs) have long been a cornerstone in deep learning, known for their capacity to model complex relationships. Recently, Kolmogorov-Arnold Networks (KANs) have emerged as a compelling alternative, utilizing highly flexible learnable activation functions directly on network edges, a departure from the neuron-centric approach of MLPs. However, KANs significantly increase the number of learnable parameters, raising concerns about their effectiveness in data-scarce environments. This paper presents a comprehensive comparative study of MLPs and KANs from both algorithmic and experimental perspectives, with a focus on low-data regimes. We introduce an effective technique for designing MLPs with unique, parameterized activation functions for each neuron, enabling a more balanced comparison with KANs. Using empirical evaluations on simulated data and two real-world data sets from medicine and engineering, we explore the trade-offs between model complexity and accuracy, with particular attention to the role of network depth. Our findings show that MLPs with individualized activation functions achieve significantly higher predictive accuracy with only a modest increase in parameters, especially when the sample size is limited to around one hundred. For example, in a three-class classification problem within additive manufacturing, MLPs achieve a median accuracy of 0.91, significantly outperforming KANs, which only reach a median accuracy of 0.53 with default hyperparameters. These results offer valuable insights into the impact of activation function selection in neural networks.

神经网络小样本学习激活函数模型比较

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。