提出新算法T-kernel SGD,高效拟合球面数据并实现最优收敛速度。
Truncated Kernel Stochastic Gradient Descent on Spheres
- 用低维投影的闭式解实现随机梯度下降,动态调节假设空间平衡偏差与方差。
- 采用常数步长即可达理论最优收敛率,克服核方法固有的饱和问题。
- 利用球多项式结构,计算复杂度降为O(n^{1+d/(d-1)ε}),适合高维球面数据。
受球谐函数结构启发,本文提出一种基于最小二乘损失的截断核随机梯度下降(T-kernel SGD)算法,用于球面数据拟合。T-kernel SGD通过在低维子空间中对随机梯度进行闭式投影,引入新颖的正则化策略,在迭代过程中动态调整假设空间,有效平衡偏差与方差。相比传统核SGD,其优势在于可使用与样本量无关的常数步长实现理论最优收敛速率,并克服核方法固有的饱和问题。此外,借助球多项式结构,我们推导出等价的T-kernel SGD,显著降低存储与计算成本:在d维球面上,仅需$\mathcal{O}(n^{1+\frac{d}{d-1}ε})$计算复杂度和$\mathcal{O}(n^{\frac{d}{d-1}ε})$存储空间即可达到最优率,其中$0<ε<\frac{1}{2}$可任意小,前提是目标函数光滑性及核积分算子特征值衰减速率满足一定正则性条件。主结果定量刻画了先验信息对收敛性的影响。数值实验验证了理论结论。
原文摘要 · Abstract (English)
Inspired by the structure of spherical harmonics, we propose the truncated kernel stochastic gradient descent (T-kernel SGD) algorithm with a least-square loss function for spherical data fitting. T-kernel SGD introduces a novel regularization strategy by implementing stochastic gradient descent through a closed-form solution of the projection of the stochastic gradient in a low-dimensional subspace. In contrast to traditional kernel SGD, the regularization strategy implemented by T-kernel SGD is more effective in balancing bias and variance by dynamically adjusting the hypothesis space during iterations. The most significant advantage of the proposed algorithm is that it can achieve theoretically optimal convergence rates using a constant step size (independent of the sample size) while overcoming the inherent saturation problem of kernel SGD. Additionally, we leverage the structure of spherical polynomials to derive an equivalent T-kernel SGD, significantly reducing storage and computational costs compared to kernel SGD. Typically, T-kernel SGD requires only $\mathcal{O}(n^{1+\frac{d}{d-1}ε})$ computational complexity and $\mathcal{O}(n^{\frac{d}{d-1}ε})$ storage to achieve optimal rates for the d-dimensional sphere, where $0<ε<\frac{1}{2}$ can be arbitrarily small if the optimal fitting or the underlying space possesses sufficient regularity. This regularity is determined by the smoothness parameter of the objective function and the decaying rate of the eigenvalues of the integral operator associated with the kernel function, both of which reflect the difficulty of the estimation problem. Our main results quantitatively characterize how this prior information influences the convergence of T-kernel SGD. The numerical experiments further validate the theoretical findings presented in this paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。