提出一种通用梯度下降法,可全局线性收敛求解任意秩矩阵的k-SVD。
$k$-SVD with Gradient Descent
- 设计简单通用的步长规则,无需问题特异性信息。
- 随机初始化下对任意k、d均实现全局线性收敛。
- 理论揭示吸引域特性,支持用Nesterov加速提升效率。
现代计算基础设施的发展推动了基于优化方法的大规模k-SVD计算研究,即求解任意秩d≥1矩阵的前k≥1个最大奇异值及其对应奇异向量。现有工作或仅适用于精确参数化(k=d)或过参数化(k>d)情形,或仅有局部收敛保证,或依赖需问题实例特定信息的步长。本文提出一种梯度下降方法,采用简单通用的步长选择规则(类似预处理),可证明对任意秩d≥1的矩阵求解k-SVD。分析表明,随机初始化下该方法对任意k,d≥1均实现全局线性收敛。进一步发现梯度法存在一个吸引域,在此区域内其行为类似赫伦法(又称巴比伦法)。该吸引域性质表明可通过Nesterov动量加速技术改进收敛速率,其性能媲美依赖Lanczos迭代等复杂方法。实验验证了理论结果。
原文摘要 · Abstract (English)
The emergence of modern compute infrastructure for iterative optimization has led to great interest in developing optimization-based approaches for a scalable computation of $k$-SVD, i.e., the $k\geq 1$ largest singular values and corresponding vectors of a matrix of rank $d \geq 1$. Despite lots of exciting recent works, all prior works fall short in this pursuit. Specifically, the existing results are either for the exact-parameterized (i.e., $k = d$) and over-parameterized (i.e., $k > d$) settings; or only establish local convergence guarantees; or use a step-size that requires problem-instance-specific oracle-provided information. In this work, we complete this pursuit by providing a gradient-descent method with a simple, universal rule for step-size selection (akin to pre-conditioning), that provably finds $k$-SVD for a matrix of any rank $d \geq 1$. We establish that the gradient method with random initialization enjoys global linear convergence for any $k, d \geq 1$. Our convergence analysis reveals that the gradient method has an attractive region, and within this attractive region, the method behaves like Heron's method (a.k.a. the Babylonian method). Our analytic results about the said attractive region imply that the gradient method can be enhanced by means of Nesterov's momentum-based acceleration technique. The resulting improved convergence rates match those of rather complicated methods typically relying on Lanczos iterations or variants thereof. We provide an empirical study to validate the theoretical results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。