标签编码方式影响神经坍缩的结构形态，揭示了分类器偏置的作用机制。
The role of class encoding in neural collapse
- 通过调节分类器偏置正则化，改变类别特征均值的几何结构
- 一热编码下特征均值从等角紧框架转为正交框架
- 适用于研究神经坍缩本质及模型设计的理论分析
神经坍缩是神经网络分类模型在训练至零分类误差后，最后一层隐藏层激活所表现出的一种结构性特征。本文基于无约束特征模型与均方误差损失函数，探讨标签编码在神经坍缩中的作用。结果表明：当标签采用一热编码且数据平衡时，随着分类器偏置正则化系数增大，每类对应的未中心化特征均值会从等角紧框架（simplex equiangular tight frame）转变为正交框架，这一结构与一热编码标签的正交特性相似。对于任意编码，我们进一步证明分类器偏置旨在使标签全局均值中心化，以补偿标签均值与原点之间的偏差。此外，本文还讨论了编码对其他神经坍缩性质的影响。
原文摘要 · Abstract (English)
Neural collapse is a structural property of the last-hidden-layer activations in neural network classification models, when trained beyond a zero classification error. In this work, we explore the role of label encoding in neural collapse by relying on the unrestricted feature model with mean squared error training loss. We demonstrate that, for one-hot encoded labels and balanced data, the uncentered mean features associated with each class transition from a simplex equiangular tight frame to an orthogonal frame when increasing the bias regularization coefficient associated with the final classifier. These structures are reminiscent of the orthogonal frame structure of one-hot encoded labels. For any arbitrary encoding, we also show that the final classifier's bias aims at centering the labels, compensating for the discrepancy between the global mean of the labels and the origin. We further discuss the role of the encoding in other neural collapse properties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。