研究不平衡数据下神经坍缩现象,揭示其几何结构与样本分布的关系。
The Exploration of Neural Collapse under Imbalanced Data
- 基于带偏置的特征模型,理论分析全局最优解的收敛特性。
- 同一类特征收敛到类均值,且行列空间呈现特定正交结构。
- 结果依赖于类别样本数的奇异值分布,适用于不平衡数据场景。
神经坍缩是训练过程中模型解的一项新发现特性。本文在数据不平衡背景下探讨神经坍缩现象,考虑含偏置项的$L$-扩展无约束特征模型,并对其全局最小值进行理论分析。研究发现:(1) 同一类内的特征收敛至该类均值,与平衡和无偏置的不平衡情形一致;(2) 几何结构由$L$个线性分类器乘积的左正交变换与类均值矩阵的右变换共同决定;(3) 左变换矩阵的部分行趋于零,其余相互正交,取决于矩阵$\ar Y = (I_K - \frac{1}{N}\mathbf{n}1_K^\top)D$的奇异值,其中$K$为类别数,$\mathbf{n}$为各类别样本数向量,$D$为对角阵且对角线元素为$\sqrt{\mathbf{n}}$;(4) 右变换矩阵的第$i$列与左变换矩阵的第$i$行对齐;(5) 给出了$\bar Y$奇异值的估计。数值实验验证了上述理论结果。
原文摘要 · Abstract (English)
Neural collapse, a newly identified characteristic, describes a property of solutions during model training. In this paper, we explore neural collapse in the context of imbalanced data. We consider the $L$-extended unconstrained feature model with a bias term and provide a theoretical analysis of global minimizer. Our findings include: (1) Features within the same class converge to their class mean, similar to both the balanced case and the imbalanced case without bias. (2) The geometric structure is mainly on the left orthonormal transformation of the product of $L$ linear classifiers and the right transformation of the class-mean matrix. (3) Some rows of the left orthonormal transformation of the product of $L$ linear classifiers collapse to zeros and others are orthogonal, which relies on the singular values of $\hat Y=(I_K-1/N\mathbf{n}1^\top_K)D$, where $K$ is class size, $\mathbf{n}$ is the vector of sample size for each class, $D$ is the diagonal matrix whose diagonal entries are given by $\sqrt{\mathbf{n}}$. Similar results are for the columns of the right orthonormal transformation of the product of class-mean matrix and $D$. (4) The $i$-th row of the left orthonormal transformation of the product of $L$ linear classifiers aligns with the $i$-th column of the right orthonormal transformation of the product of class-mean matrix and $D$. (5) We provide the estimation of singular values about $\hat Y$. Our numerical experiments support these theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。