为柯尔莫哥洛夫-阿诺德网络提供理论保障,揭示其泛化能力与模型复杂度关系。
Generalization Bounds and Model Complexity for Kolmogorov-Arnold Networks
- 基于基函数或低秩核空间构建激活函数,分析网络泛化边界
- 边界依赖系数矩阵l1范数和激活函数Lipschitz常数,不依赖节点数量
- 适用于多种回归损失函数,适合关注可解释性与理论保障的研究者
柯尔莫哥洛夫-阿诺德网络(KAN)由Liu等人(2024)提出,相比多层感知机在科学任务中具有更高可解释性和更简洁结构。本文对KAN进行严格理论分析,建立了配备特定激活函数的泛化边界:当激活函数由基函数线性组合构成时,边界与各层的算子范数适配,且仅随系数矩阵的l1范数和激活函数Lipschitz常数增长,不受节点数等组合参数影响(仅含对数因子);同时无需损失函数有界假设,适用于广泛回归型损失函数。在低秩情形下,边界随底层秩及各层激活函数的Lipschitz常数多项式增长。通过在模拟与真实数据集上训练的KAN进行数值验证,结果表明该边界具有实际意义。
原文摘要 · Abstract (English)
Kolmogorov-Arnold Network (KAN) is a network structure recently proposed by Liu et al. (2024) that offers improved interpretability and a more parsimonious design in many science-oriented tasks compared to multi-layer perceptrons. This work provides a rigorous theoretical analysis of KAN by establishing generalization bounds for KAN equipped with activation functions that are either represented by linear combinations of basis functions or lying in a low-rank Reproducing Kernel Hilbert Space (RKHS). In the first case, the generalization bound accommodates various choices of basis functions in forming the activation functions in each layer of KAN and is adapted to different operator norms at each layer. For a particular choice of operator norms, the bound scales with the $l_1$ norm of the coefficient matrices and the Lipschitz constants for the activation functions, and it has no dependence on combinatorial parameters (e.g., number of nodes) outside of logarithmic factors. Moreover, our result does not require the boundedness assumption on the loss function and, hence, is applicable to a general class of regression-type loss functions. In the low-rank case, the generalization bound scales polynomially with the underlying ranks as well as the Lipschitz constants of the activation functions in each layer. These bounds are empirically investigated for KANs trained with stochastic gradient descent on simulated and real data sets. The numerical results demonstrate the practical relevance of these bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。