arXiv:2501.00704cs.LGstat.CO2025-01被引 3

K-GAM网络用加性模型替代Transformer,参数更少且可并行。

Kolmogorov GAM Networks are all you need!

  • 基于柯尔莫哥洛夫定理构建加性模型,嵌入与目标函数解耦。
  • 参数量远低于深度学习模型,在Iris数据上表现优异。
  • 适合追求高效、低参架构的AI研究者和工业应用。

Kolmogorov GAM(K-GAM)网络被证明是一种高效且适用于训练与推理的架构。其本质为一种加性模型,其中嵌入部分独立于目标函数。该架构是柯尔莫哥洛夫超位置定理(KST)在机器学习中的体现,可高效表示多变量函数,适用于编码字典(即“查表”)。KST还提供基于柯本函数平移的表示形式。本文旨在将这一理论转化为机器学习应用场景,实现人工智能中的有效建模。所提架构等价于与函数无关的拓扑嵌入结合加性层,采用广义加性模型(GAM),形成一类参数显著少于当前深度学习算法的学习方法。该方法支持并行化实现,计算效率高。我们以统计学习中的Iris数据为例展示方法有效性。结果表明,非线性嵌入的加性模型可作为变压器架构的统计学替代——后者本质上是核平滑器。因此,加性KAN模型为变换器提供了自然替代方案。最后,文章展望了未来研究方向。

原文摘要 · Abstract (English)

Kolmogorov GAM (K-GAM) networks are shown to be an efficient architecture for training and inference. They are an additive model with an embedding that is independent of the function of interest. They provide an alternative to the transformer architecture. They are the machine learning version of Kolmogorov's Superposition Theorem (KST) which provides an efficient representations of a multivariate function. Such representations have use in machine learning for encoding dictionaries (a.k.a. "look-up" tables). KST theory also provides a representation based on translates of the Köppen function. The goal of our paper is to interpret this representation in a machine learning context for applications in Artificial Intelligence (AI). Our architecture is equivalent to a topological embedding which is independent of the function together with an additive layer that uses a Generalized Additive Model (GAM). This provides a class of learning procedures with far fewer parameters than current deep learning algorithms. Implementation can be parallelizable which makes our algorithms computationally attractive. To illustrate our methodology, we use the Iris data from statistical learning. We also show that our additive model with non-linear embedding provides an alternative to transformer architectures which from a statistical viewpoint are kernel smoothers. Additive KAN models therefore provide a natural alternative to transformers. Finally, we conclude with directions for future research.

神经网络加性模型高效架构机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。