深度线性网络可自动适应数据隐含结构,实现逆问题求解的全局收敛。
Solving Inverse Problems with Deep Linear Neural Networks: Global Convergence Guarantees for Gradient Descent with Weight Decay
- 用带权重衰减的梯度下降训练过参数深度线性网络
- 在合理步长和初始化下收敛到近似最优解映射
- 适合研究神经网络泛化与收敛理论的研究者
机器学习方法常用于求解逆问题,即从少量间接测量中估计未知信号。尽管神经网络在实践中表现良好,但理论保障有限。本文研究一个欠定线性逆问题,其解映射可能不唯一。传统方法(如压缩感知)通过假设源信号存在潜在低维结构来保证解的唯一性。本文探讨:以梯度下降加权重衰减训练的深度线性网络,能否自动适应未知的低维结构?我们证明,在适度过参数化条件下,这类网络能收敛到一个近似解映射,准确解决逆问题,并隐式编码数据的潜在子空间结构。在实际步长和初始化方案下,权重衰减使网络自动适应数据中的隐含子空间结构。研究揭示:正则化与过参数化提升泛化能力,而过参数化还加速了训练过程的收敛。
原文摘要 · Abstract (English)
Machine learning methods are commonly used to solve inverse problems, wherein an unknown signal must be estimated from few indirect measurements generated via a known acquisition procedure. In particular, neural networks perform well empirically but have limited theoretical guarantees. In this work, we study an underdetermined linear inverse problem that admits several possible solution operators that map measurements to estimates of the target signal. A standard remedy (e.g., in compressed sensing) for establishing the uniqueness of the solution mapping is to assume the existence of a latent low-dimensional structure in the source signal. We ask the following question: do deep linear neural networks adapt to unknown low-dimensional structure when trained by gradient descent with weight decay regularization? We prove that mildly overparameterized deep linear networks trained in this manner converge to an approximate solution mapping that accurately solves the inverse problem while implicitly encoding latent subspace structure. We show rigorously that deep linear networks trained with weight decay automatically adapt to latent subspace structure in the data under practical stepsize and weight initialization schemes. Our work highlights that regularization and overparameterization improve generalization, while overparameterization also accelerates convergence during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。