解释深度网络梯度为何有隐式偏置，发现奇异值指数增长与对齐机制。
Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment
- 从分段线性网络视角出发，分析雅可比矩阵的奇异值动态
- 揭示深度导致奇异值指数级增长与谱分离现象
- 适合研究深度学习隐式偏置机制的学者参考
理解深度网络中基于梯度训练的强隐式偏差仍具挑战性，主要因可解析的奇异值动态通常仅存在于平衡的深层线性模型中。本文提出新路径：基于深度雅可比矩阵的两个理论可验证特征——有序奇异值的深度诱导指数增长和强谱分离。通过固定门控视角，将分段线性网络的雅可比简化为单个激活区域内掩码线性映射的乘积，证明了初始状态下主导奇异值受李亚普诺夫指数支配，并在可解析掩码模型中给出闭式表达，量化有限深度修正。进一步表明，足够强的分离会迫使矩阵乘积中的奇异向量对齐，使中间雅可比近似共享奇异基。这些结果共同支持一个近似框架，使奇异值动态有效解耦，无需平衡假设即可类比经典平衡深层线性分析。固定门控设置下的实验验证了预测的缩放、对齐及相应动态，支持了低秩雅可比结构作为隐式偏置驱动力的机制解释。
原文摘要 · Abstract (English)
Understanding why gradient-based training in deep networks exhibits strong implicit bias remains challenging, in part because tractable singular-value dynamics are typically available only for balanced deep linear models. We propose an alternative route based on two theoretically grounded and empirically testable signatures of deep Jacobians: depth-induced exponential scaling of ordered singular values and strong spectral separation. Adopting a fixed-gates view of piecewise-linear networks, where Jacobians reduce to products of masked linear maps within a single activation region, we prove the existence of Lyapunov exponents governing the top singular values at initialization, give closed-form expressions in a tractable masked model, and quantify finite-depth corrections. We further show that sufficiently strong separation forces singular-vector alignment in matrix products, yielding an approximately shared singular basis for intermediate Jacobians. Together, these results motivate an approximation regime in which singular-value dynamics become effectively decoupled, mirroring classical balanced deep-linear analyses without requiring balancing. Experiments in fixed-gates settings validate the predicted scaling, alignment, and resulting dynamics, supporting a mechanistic account of emergent low-rank Jacobian structure as a driver of implicit bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。