提出自然黎曼梯度法，高效优化函数张量网络模型。
Natural Riemannian gradient for learning functional tensor networks
- 基于自然梯度思想，设计与基底无关的优化方向。
- 在分类数据集上收敛速度显著优于传统方法。
- 适用于多种张量网络表示，适合高维学习任务研究者。
我们研究使用低秩函数树状张量网络（TTN）作为学习模型的机器学习任务。虽然在最小二乘回归中可通过交替优化高效求解，但在其他问题如多项式逻辑回归中则不可行。本文提出一种适用于任意损失函数的自然黎曼梯度下降方法，该方法基于Amari提出的自然梯度理论，其搜索方向独立于底层函数张量积空间的基底选择。该框架同时适用于因子化和流形表示两种函数TTN建模方式。为实际应用，我们设计了一套层次化的高效近似方法以计算参数空间中的更新。数值实验在常见分类数据集上验证了理论结果，表明采用自然黎曼梯度下降能显著改善收敛行为，相比标准黎曼梯度方法有明显优势。
原文摘要 · Abstract (English)
We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model. While in the case of least-squares regression, low-rank functional TTNs can be efficiently optimized using alternating optimization, this is not directly possible in other problems, such as multinomial logistic regression. We propose a natural Riemannian gradient descent type approach applicable to arbitrary losses which is based on the natural gradient by Amari. In particular, the search direction obtained by the natural gradient is independent of the choice of basis of the underlying functional tensor product space. Our framework applies to both the factorized and manifold-based approach for representing the functional TTN. For practical application, we propose a hierarchy of efficient approximations to the true natural Riemannian gradient for computing the updates in the parameter space. Numerical experiments confirm our theoretical findings on common classification datasets and show that using natural Riemannian gradient descent for learning considerably improves convergence behavior when compared to standard Riemannian gradient methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。