基于数据几何设计的初始化方法,让神经网络从更优起点开始学习。
S-GAI: Spectral Geometry-Aware Initialization for Sigmoidal MLPs -- From Dataset Geometry to Network Weights

- 根据图像数据的类别谱几何特征,用SVD提取主方向与尺度
- 用双门限激活单元构建共享隐藏层,直接映射数据分布结构
- 冻结隐藏层仍优于随机初始化,证明几何信息有效注入
经典通用逼近定理证明了符号函数多层感知机的表达能力,但未说明初始权重如何编码数据分布几何。本文提出S-GAI框架,为单隐层符号函数MLP设计谱几何感知初始化。基于符号单元可作为平滑半空间门的构造思想,从人工设定的平面几何转向从图像数据中估计的类别谱几何。对每类数据,通过SVD获得均值、主方向和谱尺度;设定能量阈值保留关键方向,并用两个符号门表示每个方向。这些类别特异性门构成共享隐层,直接由训练集初始化。同时提出基于SVD的子空间分类器作为非神经几何参考,检验估计的谱类别几何是否已具备判别能力。在MNIST、Fashion-MNIST及更具挑战性的CIFAR-10上的实验表明,经S-GAI初始化的MLP初始隐状态显著更丰富,全量训练下达到与Xavier初始化相当的最终准确率。当隐层冻结仅训练输出层时,性能仍优于冻结随机门,证实S-GAI成功将类别谱几何嵌入MLP。
原文摘要 · Abstract (English)
Classical universal approximation theorems establish the expressive power of sigmoidal multilayer perceptrons, but they do not prescribe how initial weights should encode the geometry of a data distribution. We propose S-GAI, a spectral geometry-aware initialization framework for one-hidden-layer sigmoidal MLPs. Starting from the constructive idea that sigmoid units can act as smooth half-space gates, we move from hand-specified planar geometry to class-wise spectral geometry estimated from image data. For each class, SVD provides a mean, principal directions, and spectral scales. An energy threshold selects the retained directions, and each retained direction is represented by two sigmoid gates. These class-specific gates form a shared hidden layer initialized directly from the training set. We also formulate a SVD-based subspace classifier as a non-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP. Experiments on MNIST, Fashion-MNIST, and a more challenging CIFAR-10 test show that the S-GAI-initialized MLP starts from a substantially more informative hidden state than Xavier initialization and reaches comparable final accuracy under full training. When the hidden layer is frozen, training only the output layer still gives stronger performance than frozen random gates, providing evidence that S-GAI effectively embeds class-wise spectral geometry into the MLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。