发现深度回归模型各层都存在结构坍缩，揭示了深层网络学习的简洁规律。
Deep Neural Regression Collapse
- 首次证明回归模型在多层均出现特征坍缩现象
- 坍缩层特征子空间与目标维度对齐，预测误差接近整体误差
- 适合研究深层网络内在结构与正则化机制的人参考
神经坍缩现象有助于揭示深度分类器中的稀疏与低秩结构。近期工作将神经坍缩定义扩展至回归问题，但仅在最后一层进行测量。本文首次证明，神经回归坍缩（NRC）在不同模型的多个层级中均存在。在坍缩层中，特征位于对应目标维度的子空间内，特征协方差与目标协方差对齐，层权重的输入子空间与特征子空间一致，且特征的线性预测误差接近模型整体预测误差。此外，我们表明具备深度坍缩的模型能学习到低秩目标的内在维度，并探讨了权重衰减在诱导深度坍缩中的必要性。本工作为深度网络在回归任务中所学结构提供了更完整的图景。
原文摘要 · Abstract (English)
Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。