通过学习权重空间几何结构，用更少参数实现与全训练模型相当的性能。
HalfNet: Randomized Neural Networks with Learned Subspace Geometry
- 从可学习低秩协方差分布中随机采样权重，替代固定分布
- 在MNIST和CIFAR-10上以极少参数达到全训练模型精度
- 揭示权重空间几何比参数值更重要，适合资源受限场景
许多研究尝试将神经网络部分权重固定为从特定分布（如$N(0, I)$）随机采样的值。本文提出的HalfNet从$N(0, Σ)$中采样，其中协方差矩阵$Σ$具有低秩分解，其几何结构由数据学习得到。在MNIST和CIFAR-10上的实验表明，HalfNet在使用显著更少参数的情况下，可达到全训练多层感知机的性能。谱分析显示，神经网络的预测能力主要源于权重空间的几何结构，而非单个参数的精确值；我们观察到准确率随秩平滑变化。HalfNet并非针对低秩结构的架构技巧，而是一种数据相关的随机嵌入，也可通过监督度量学习、随机特征或核方法视角理解。
原文摘要 · Abstract (English)
Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.g., $N(0, I)$. Our proposed HalfNet draws random weights from $N(0, Σ)$, where $Σ$, which defines the geometry of the distribution, has a low-rank factorization that we learn from data. Experiments on MNIST and CIFAR-10 demonstrate that HalfNet can match the performance of fully trained multilayer perceptrons while using substantially fewer parameters. Spectral analysis indicates that much of the predictive power of neural networks lies in the geometry of their weight space rather than in the precise values of individual parameters, and we observe that accuracy scales smoothly with rank. HalfNet is not a neural architecture trick for low-rank structure; it implements a data-dependent random embedding that can also be interpreted through supervised metric learning, or random-feature and kernel perspectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。