深度决定谱结构，非数据分布影响优化难易。
Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
- 分析深层线性网络，发现即使数据平衡仍存在显著谱分裂。
- 主导与主体特征值之比随网络深度线性增长。
- 提醒优化算法设计需兼顾模型结构与数据特性。
Hessian 矩阵的特征值分布对理解深度神经网络的优化景观至关重要。以往研究将广为人知的「体块-尖峰」谱结构（少数主导特征值与一大群小特征值分离）归因于数据协方差矩阵的不平衡。本文挑战这一观点，证明该谱分裂可完全由网络架构引发，与数据不平衡无关。具体而言，在深层线性网络设定下，即便数据协方差完全平衡，Hessian 依然呈现分裂结构：一个主导特征值簇与一个主体特征值簇。关键发现是，主导与主体特征值的比值随网络深度呈线性增长。这表明谱间隙主要受模型架构影响，而非仅由数据分布决定。结果提示，设计深度网络优化算法时，应同时考虑模型架构与数据特性。
原文摘要 · Abstract (English)
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。