通过强制神经网络各层形成等角紧框架,大幅减少参数量且不损失性能。
Leveraging Intermediate Neural Collapse with Simplex ETFs for Efficient Deep Neural Networks
- 在有效深度后的所有层施加等角紧框架约束,实现参数高效训练。
- 相比基线模型,参数量减少超60%,测试准确率保持相当。
- 适用于需要轻量化部署的深度网络,如移动端或边缘设备。
神经坍缩是神经网络训练末期出现的现象,表现为网络激活、类别均值和线性分类器权重收敛至单纯形等角紧框架(ETF),该结构能最大化子空间内向量间的相互距离。已有研究表明,将网络最后一层约束为ETF可显著减少可训练参数而不降低准确率。本文进一步发现,深层全连接网络在超过特定有效深度后,各层均呈现神经坍缩现象。基于此,提出两种新方法:Adaptive-ETF 在有效深度后的所有层施加ETF约束;ETF-Transformer 则在Transformer块的前馈层中引入ETF约束。实验表明,二者在训练与测试性能上与基线模型相当,但可学习参数数量大幅减少。
原文摘要 · Abstract (English)
Neural collapse is a phenomenon observed during the terminal phase of neural network training, characterized by the convergence of network activations, class means, and linear classifier weights to a simplex equiangular tight frame (ETF), a configuration of vectors that maximizes mutual distance within a subspace. This phenomenon has been linked to improved interpretability, robustness, and generalization in neural networks. However, its potential to guide neural network training and regularization remains underexplored. Previous research has demonstrated that constraining the final layer of a neural network to a simplex ETF can reduce the number of trainable parameters without sacrificing model accuracy. Furthermore, deep fully connected networks exhibit neural collapse not only in the final layer but across all layers beyond a specific effective depth. Using these insights, we propose two novel training approaches: Adaptive-ETF, a generalized framework that enforces simplex ETF constraints on all layers beyond the effective depth, and ETF-Transformer, which applies simplex ETF constraints to the feedforward layers within transformer blocks. We show that these approaches achieve training and testing performance comparable to those of their baseline counterparts while significantly reducing the number of learnable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。