发现特征均值范数达到临界值时，神经网络将发生表征坍缩。
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
- 用特征范数是否突破临界值来预测网络表征坍缩的时机。
- 在不同模型和数据集上，该临界值稳定且受训练条件影响小。
- 深度、激活函数、正则化等结构因素共同决定坍缩速度与临界值。
神经坍缩（NC）——即倒数第二层特征收敛至等角紧框架——在平衡态下已有较好理解，但其触发动态仍不清晰。本文发现一个简单且可预测的规律：当特征均值范数达到模型-数据集特异的临界值 fn* 时，NC 即发生；该值在每对（模型, 数据集）中高度集中（变异系数 CV < 8%），训练条件主要影响趋近速率而非值本身。标准训练轨迹中，特征范数低于 fn* 的时间点始终先于 NC 发生，提供实用预测工具，平均领先 62 个周期（平均绝对误差 24）。直接干预实验确认 fn* 是梯度流的稳定吸引子——特征尺度扰动会被自我修正，最终收敛至同一值（p > 0.2）。完成（架构）×（数据集）网格分析后，最显著结果为：在 MNIST 上，ResNet-20 的 fn* = 5.867，相比仅 +68% 的 CIFAR-10，架构效应高达 +458%。该网格呈现强非加性特征，fn* 无法分解为独立的架构与数据贡献。四个结构性规律浮现：(1) 深度对坍缩速度有非单调影响；(2) 激活函数同时决定坍缩速度与 fn*；(3) 权重衰减构成三阶段相图——过少则慢，最优范围最快，过多则抑制坍缩；(4) 宽度单调加速坍缩，但仅使 fn* 偏移 ≤13%。这些结果确立特征范数动态为预测 NC 时机的可操作诊断手段，提示范数阈值行为是深层网络延迟表征重组的普遍机制。
原文摘要 · Abstract (English)
Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。