LeanKAN用更少参数实现更强性能,是KAN网络的高效替代方案。
LeanKAN: A Parameter-Lean Kolmogorov-Arnold Network Layer with Improved Memory Efficiency and Convergence Behavior
- 通过简化结构和减少冗余激活,实现参数精简
- 在微分方程等任务中表现优于更大规模的MultKAN
- 适用于传统KAN及KAN-ODE、DeepOKAN等复杂架构
最近提出的科尔莫戈罗夫-阿诺德网络(KAN)是数据驱动建模对多层感知机(MLP)的一种有前景替代。虽然原始KAN层仅能表示加法运算,新提出的MultKAN层通过引入加法与乘法子节点以提升表达能力。然而,我们发现MultKAN存在输出层适用性受限、参数量庞大且含多余激活、超参数复杂等问题。为此,我们提出LeanKAN,作为MultKAN和传统AddKAN层的直接、模块化替代。LeanKAN通过支持通用输出层、显著降低给定结构下的参数量以及减少超参数数量,解决上述三大缺陷。作为标准AddKAN和MultKAN层的一一替换,LeanKAN不仅可应用于传统KAN学习问题,还可作为骨干用于增强型KAN结构如KAN-ODE或DeepOKAN。我们在标准玩具问题及基于KAN-ODE的常微分方程、偏微分方程学习任务中验证了其简洁性与高效性,结果表明其稀疏参数化与紧凑结构提升了表达能力和学习能力,在多项任务中超越相似甚至更大规模的MultKAN。
原文摘要 · Abstract (English)
The recently proposed Kolmogorov-Arnold network (KAN) is a promising alternative to multi-layer perceptrons (MLPs) for data-driven modeling. While original KAN layers were only capable of representing the addition operator, the recently-proposed MultKAN layer combines addition and multiplication subnodes in an effort to improve representation performance. Here, we find that MultKAN layers suffer from a few key drawbacks including limited applicability in output layers, bulky parameterizations with extraneous activations, and the inclusion of complex hyperparameters. To address these issues, we propose LeanKANs, a direct and modular replacement for MultKAN and traditional AddKAN layers. LeanKANs address these three drawbacks of MultKAN through general applicability as output layers, significantly reduced parameter counts for a given network structure, and a smaller set of hyperparameters. As a one-to-one layer replacement for standard AddKAN and MultKAN layers, LeanKAN is able to provide these benefits to traditional KAN learning problems as well as augmented KAN structures in which it serves as the backbone, such as KAN Ordinary Differential Equations (KAN-ODEs) or Deep Operator KANs (DeepOKAN). We demonstrate LeanKAN's simplicity and efficiency in a series of demonstrations carried out across a standard KAN toy problem as well as ordinary and partial differential equations learned via KAN-ODEs, where we find that its sparser parameterization and compact structure serve to increase its expressivity and learning capability, leading it to outperform similar and even much larger MultKANs in various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。