KAN网络训练的优化、泛化与隐私边界首次被系统分析。
Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networks
- 基于梯度下降分析两层KAN,推导出训练动态与泛化上界。
- 在逻辑损失下,网络宽度仅需多对数级即可实现1/T优化率和1/n泛化率。
- 揭示私有训练中多对数宽度必要性,指导实际宽度选择与早停策略。
Kolmogorov-Arnold网络(KAN)作为标准MLP的结构化替代方案近年来兴起,但其训练动态、泛化能力及隐私性质仍缺乏严谨理论。本文分析两层KAN的梯度下降(GD)训练,推导出涵盖训练动态、泛化与差分隐私(DP)效用的通用界。以逻辑损失在NTK可分假设下为例,证明多对数级网络宽度足以使GD达到1/T的优化率和1/n的泛化率(其中T为迭代次数,n为样本量)。在私有设定下,刻画了(ε,δ)-DP所需噪声,并获得√d/(nε)的效用界(d为输入维度),与一般凸利普希茨问题的经典下界一致。结果表明,多对数宽度在私有训练中不仅是充分条件,更是必要条件,揭示了非私有与私有训练范式间的定性差异。实验验证理论洞察对网络宽度选择与早停的实际指导价值。
原文摘要 · Abstract (English)
Kolmogorov--Arnold Networks (KANs) have recently emerged as a structured alternative to standard MLPs, yet a principled theory for their training dynamics, generalization, and privacy properties remains limited. In this paper, we analyze gradient descent (GD) for training two-layer KANs and derive general bounds that characterize their training dynamics, generalization, and utility under differential privacy (DP). As a concrete instantiation, we specialize our analysis to logistic loss under an NTK-separable assumption, where we show that polylogarithmic network width suffices for GD to achieve an optimization rate of order $1/T$ and a generalization rate of order $1/n$, with $T$ denoting the number of GD iterations and $n$ the sample size. In the private setting, we characterize the noise required for $(ε,δ)$-DP and obtain a utility bound of order $\sqrt{d}/(nε)$ (with $d$ the input dimension), matching the classical lower bound for general convex Lipschitz problems. Our results imply that polylogarithmic width is not only sufficient but also necessary under differential privacy, revealing a qualitative gap between non-private (sufficiency only) and private (necessity also emerges) training regimes. Experiments further illustrate how these theoretical insights can guide practical choices, including network width selection and early stopping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。