arXiv:2504.09708math.OCcs.LG2025-04NeurIPS被引 49

针对过参数化矩阵分解收敛慢问题,提出高效预处理梯度下降法。

Preconditioned Gradient Descent for Over-Parameterized Nonconvex Matrix Factorization

  • 通过特定范围的L2正则化修复奇异问题,恢复线性收敛率。
  • 在过参数化情况下仍实现线性收敛,误差达信息论最优边界。
  • 算法对真实数据病态不敏感,适合噪声环境与多种变体。

实际非凸矩阵分解中,真实解的秩 $r^{/star}$ 常未知,模型秩 $r$ 常被高估为 $r>r^{/star}$。这种过参数化情形使局部搜索算法收敛速度从 $r=r^{/star}$ 时的线性速率降至亚线性。本文提出一种低成本预处理器,用于矩阵感知的非凸矩阵分解,使梯度下降在 $r>r^{/star}$ 时仍恢复线性收敛速率,并对真实数据可能存在的病态不敏感。经典梯度下降在解附近因模型因子趋于奇异而变慢。关键发现是:通过 $\ ext{L}_2$ 正则化并选择合适阻尼参数可纠正此奇异。该参数可由当前迭代点廉价估计。所提算法——预处理梯度下降(PrecGD)——在噪声下稳定,线性收敛至信息论最优误差界。数值实验表明,PrecGD在其他非凸矩阵分解变体中同样有效恢复过参数化下的线性收敛。

原文摘要 · Abstract (English)

In practical instances of nonconvex matrix factorization, the rank of the true solution $r^{\star}$ is often unknown, so the rank $r$ of the model can be overspecified as $r>r^{\star}$. This over-parameterized regime of matrix factorization significantly slows down the convergence of local search algorithms, from a linear rate with $r=r^{\star}$ to a sublinear rate when $r>r^{\star}$. We propose an inexpensive preconditioner for the matrix sensing variant of nonconvex matrix factorization that restores the convergence rate of gradient descent back to linear, even in the over-parameterized case, while also making it agnostic to possible ill-conditioning in the ground truth. Classical gradient descent in a neighborhood of the solution slows down due to the need for the model matrix factor to become singular. Our key result is that this singularity can be corrected by $\ell_{2}$ regularization with a specific range of values for the damping parameter. In fact, a good damping parameter can be inexpensively estimated from the current iterate. The resulting algorithm, which we call preconditioned gradient descent or PrecGD, is stable under noise, and converges linearly to an information theoretically optimal error bound. Our numerical experiments find that PrecGD works equally well in restoring the linear convergence of other variants of nonconvex matrix factorization in the over-parameterized regime.

矩阵分解梯度下降过参数化优化加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。