用图示展开分析大规模学习梯度流，揭示不同学习阶段并求解非线性方程。
Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions
- 通过类费曼图的级数展开描述损失演化，系数由图形编码。
- 在高阶张量CP分解中发现自由演化、NTK等多个学习相，依赖参数缩放与对称性。
- 将展开式化为一阶偏微分方程，可用特征线法求解，理论与实验高度吻合。
我们建立了一个通用数学框架，用于分析大规模学习问题中的尺度规律，并推导梯度流（GF）的显式解析解。核心创新在于对损失演化进行形式幂级数展开，其系数以类似费曼图的图形表示。我们证明该展开在大尺寸极限下有良好定义，可揭示不同学习相，某些情况下可获得非线性梯度流的显式解。研究聚焦于高阶张量的典型秩分解（CP）学习，发现存在多个截然不同的极端懒惰与丰富梯度流相，如自由演化、神经正切核（NTK）以及欠/过参数化均值场。这些相态取决于参数缩放、张量阶数及模型对称性，具有特定而微妙的依赖关系。此外，我们提出一种通用方法，通过将形式损失展开转化为偏微分方程（PDE）来求和；在广泛场景中，该方程为一阶，可用特征线法求解。理论预测与实验结果表现出极佳一致性。
原文摘要 · Abstract (English)
We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation is a formal power series expansion of the loss evolution, with coefficients encoded by diagrams akin to Feynman diagrams. We show that this expansion has a well-defined large-size limit that can be used to reveal different learning phases and, in some cases, to obtain explicit solutions of the nonlinear GF. We focus on learning Canonical Polyadic (CP) decompositions of high-order tensors, and show that this model has several distinct extreme lazy and rich GF regimes such as free evolution, NTK and under- and over-parameterized mean-field. We show that these regimes depend on the parameter scaling, tensor order, and symmetry of the model in a specific and subtle way. Moreover, we propose a general approach to summing the formal loss expansion by reducing it to a PDE; in a wide range of scenarios, it turns out to be first-order and solvable by the method of characteristics. We observe a very good agreement of our theoretical predictions with experimental results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。