arXiv:2411.01248cs.LG2024-11NeurIPS被引 16

让神经网络更快收敛,通过强制特征逼近最优对称结构。

Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame

  • 在训练中每步优化特征分布,使其逼近最简对称框架。
  • 实验显示收敛速度提升,且训练更稳定,不依赖具体模型。
  • 适合追求快速训练和鲁棒性的分类任务研究者使用。

神经坍缩(Neural Collapse, NC)是近期发现的神经网络现象,表明当网络训练至零训练损失时,最终分类层权重会收敛到等角紧框架(Simplex Equiangular Tight Frame, ETF),该结构能最大化不同类别权重间的分离度。根据对偶性,倒数第二层的特征均值也趋于同一ETF。由于这种对称结构为最优解,本文提出利用该性质加速收敛。具体地,我们定义任意训练阶段倒数第二层特征的最近ETF几何,并将其建模为黎曼优化问题。每轮迭代中,通过求解此内层优化,隐式将分类器权重设为最近的ETF,该过程封装为声明式节点以支持反向传播。在合成数据及真实世界架构上的分类任务实验表明,该方法显著加快收敛并提升训练稳定性。

原文摘要 · Abstract (English)

Neural Collapse (NC) is a recently observed phenomenon in neural networks that characterises the solution space of the final classifier layer when trained until zero training loss. Specifically, NC suggests that the final classifier layer converges to a Simplex Equiangular Tight Frame (ETF), which maximally separates the weights corresponding to each class. By duality, the penultimate layer feature means also converge to the same simplex ETF. Since this simple symmetric structure is optimal, our idea is to utilise this property to improve convergence speed. Specifically, we introduce the notion of nearest simplex ETF geometry for the penultimate layer features at any given training iteration, by formulating it as a Riemannian optimisation. Then, at each iteration, the classifier weights are implicitly set to the nearest simplex ETF by solving this inner-optimisation, which is encapsulated within a declarative node to allow backpropagation. Our experiments on synthetic and real-world architectures for classification tasks demonstrate that our approach accelerates convergence and enhances training stability.

神经坍缩优化算法分类任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。