用线性视角揭示ReLU网络训练中的分类机制
Unveiling the Training Dynamics of ReLU Networks through a Linear Lens
- 将多层ReLU网络转为输入相关的单层线性模型
- 同类别样本的有效权重逐渐收敛，异类则发散
- 适合研究深度学习内部表征与决策边界
深度神经网络，尤其是采用修正线性单元（ReLU）的网络，通常被视为复杂、高维、非线性的系统，这给理解其内部学习机制带来了巨大挑战。本文提出一种新的分析框架，将多层ReLU网络重构为一个具有输入依赖‘有效权重’的等效单层线性模型。对于任一输入样本，其ReLU单元的激活模式会形成一条独特的计算路径，相当于在权重中置零一部分。通过组合各层的激活权重，可得到一个映射该样本输入到输出的有效权重矩阵 $W_{\text{eff}}(x)$。我们提出，这些有效权重的演化揭示了表征学习的基本规律。实验表明，随着训练进行，同一类样本的有效权重趋于收敛，而不同类样本则趋向分离。通过追踪这些样本级有效权重的轨迹，我们提供了一个新视角，用以解释类别特定决策边界的形成及语义表示的涌现。
原文摘要 · Abstract (English)
Deep neural networks, particularly those employing Rectified Linear Units (ReLU), are often perceived as complex, high-dimensional, non-linear systems. This complexity poses a significant challenge to understanding their internal learning mechanisms. In this work, we propose a novel analytical framework that recasts a multi-layer ReLU network into an equivalent single-layer linear model with input-dependent "effective weights". For any given input sample, the activation pattern of ReLU units creates a unique computational path, effectively zeroing out a subset of weights in the network. By composing the active weights across all layers, we can derive an effective weight matrix, $W_{\text{eff}}(x)$, that maps the input directly to the output for that specific sample. We posit that the evolution of these effective weights reveals fundamental principles of representation learning. Our work demonstrates that as training progresses, the effective weights corresponding to samples from the same class converge, while those from different classes diverge. By tracking the trajectories of these sample-wise effective weights, we provide a new lens through which to interpret the formation of class-specific decision boundaries and the emergence of semantic representations within the network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。