用几何理论揭示深度线性网络正则化如何导致平衡状态
Regularization Implies balancedness in the deep linear network
- 基于几何不变量理论，证明L2正则化使网络趋于平衡流形
- 正则化流以统一指数速率收敛到平衡状态，平方动量映射流全局收敛
- 揭示训练过程由正则化流与学习流共同驱动，适用于模型简化与贝叶斯分析
我们运用几何不变量理论（GIT）研究深度线性网络（DLN）。利用Kempf-Ness定理，证明L²正则化在平衡流形上取得最小值。引入基于纤维黎曼几何的平衡流，发现由L²正则化定义的平衡流以统一指数速率收敛至平衡流形；由平方动量映射定义的平衡流可显式计算，并证明其全局收敛。该框架将训练动态分解为两个独立梯度流：纤维上的正则化流与平衡流形上的学习流。同时为深度学习与线性系统理论中的平衡性提供统一数学基础，并从快慢系统、模型降维及贝叶斯原理角度解释平衡性。
原文摘要 · Abstract (English)
We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the $L^2$ regularizer is minimized on the balanced manifold. We introduce related balancing flows using the Riemannian geometry of fibers. The balancing flow defined by the $L^2$ regularizer is shown to converge to the balanced manifold at a uniform exponential rate. The balancing flow defined by the squared moment map is computed explicitly and shown to converge globally. This framework allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. It also provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of fast-slow systems, model reduction and Bayesian principles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。