用权重格拉姆矩阵揭示深度网络特征线性化过程
The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
- 提出特征学习方程，用权重格拉姆矩阵刻画特征演化
- 发现深层网络逐步将特征对齐到目标线性结构
- 解释神经坍缩等现象，适合研究模型训练机制者
理解深度神经网络如何学习表征仍是机器学习理论的核心挑战。本文提出一种以特征为中心的分析框架，通过将权重更新与特征演化关联起来，引入一个简洁恒等式——特征学习方程，指出权重格拉姆矩阵是捕捉特征动态的关键对象。由此可将梯度下降解释为隐式诱导特征的假设演化，其协方差结构（称虚拟协方差）刻画了训练过程中表征的演变。基于此，我们提出目标线性度（Target Linearity），量化特征与目标间的线性对齐程度。通过分析训练过程和逐层动态，我们表明深度网络会逐步将表征转换为目标线性结构。这一线性化视角统一解释了若干经验现象，包括神经坍缩（Neural Collapse）和生成模型中的线性插值。
原文摘要 · Abstract (English)
Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。