无需逆矩阵的流形自然梯度法，提升优化效率与约束处理能力。
Inversion-Free Natural Gradient Descent on Riemannian Manifolds
- 在流形上直接构造逆FIM近似，避免显式求逆
- 收敛速度达O(log s / s^α)，大模型可用低存储变体
- 适合带约束、非可识别参数的统计建模场景
自然梯度法是统计优化的核心工具，但其应用受限于欧氏参数空间假设、费希尔信息矩阵（FIM）重复估计及后续求逆的计算开销。本文提出一种内在的、无需求逆的自然梯度方法，适用于参数位于一般黎曼流形上的统计模型。该框架可自然施加参数约束、消除不可识别参数，并利用测地凸性。算法基于对逆FIM的动态近似，该近似在流形上直接维护，并利用低秩矩阵恒等式高效更新新得分向量。证明了迭代序列几乎必然收敛速度为O(log s / s^α)，近似FIM也具有类似收敛率。进一步提出有限内存变体，实现次二次存储复杂度，适用于大规模应用。在Bures-Wasserstein流形上的变分贝叶斯、Stiefel流形上的归一化流以及降秩逻辑回归中验证了方法的有效性。
原文摘要 · Abstract (English)
The natural gradient method is a central tool for statistical optimisation, but its broader application is hindered by the assumption of a Euclidean parameter space, the repeated estimation of the Fisher information matrix (FIM), and the computational cost of its subsequent inversion. This paper proposes an intrinsic, inversion-free natural gradient method for statistical models whose parameters lie on general Riemannian manifolds. Formulating statistical optimisation in this non-Euclidean setting allows for the natural enforcement of parameter constraints, the elimination of non-identifiable parameters, and the exploitation of geodesic convexity. Our algorithm is based on a moving approximation of the inverse FIM, which is maintained directly on the manifold. This approximation is efficiently updated with new score vectors using low-rank matrix identities. We prove almost-sure convergence rates of $O(\log s / s^α)$ for the sequence of iterates, and a similar rate for the approximate FIM. A limited-memory variant with sub-quadratic storage complexity is further proposed for large-scale applications. We demonstrate the efficacy of our method on variational Bayes within the Bures-Wasserstein manifold, normalising flows on the Stiefel manifold, and reduced-rank logistic regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。