提出高效求解深层网络海森矩阵逆向量乘积的方法
The Hessian of tall-skinny networks is easy to invert
- 直接计算海森逆向量乘积，无需存储海森矩阵及其逆
- 时间与存储复杂度仅随层数线性增长，远低于传统方法
- 适用于需要快速二阶优化的深度学习场景
我们提出一种精确算法，用于求解形如 $Hx=b$ 的线性系统，其中 $H$ 是深度网络的海森矩阵。该方法在不存储海森矩阵或其逆矩阵的前提下，计算海森逆向量乘积。其时间和存储开销均与网络层数呈线性关系。相比之下，传统先计算海森矩阵再求解的方法，时间和存储复杂度分别与层数的平方和立方成正比。该方法的复杂度大致相当于 Pearlmutter 的海森向量乘积算法。
原文摘要 · Abstract (English)
We describe an exact algorithm to solve linear systems of the form $Hx=b$ where $H$ is the Hessian of a deep net. The method computes Hessian-inverse-vector products without storing the Hessian or its inverse. It requires time and storage that scale linearly in the number of layers. This is in contrast to the naive approach of first computing the Hessian, then solving the linear system, which takes storage and time that are respectively quadratic and cubic in the number of layers. The Hessian-inverse-vector product method scales roughly like Pearlmutter's algorithm for computing Hessian-vector products.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。