用流形几何分析神经网络空间，发现高性能网络聚集在一起。
Exploring the Manifold of Neural Networks Using Diffusion Geometry
- 以隐藏层表示距离构建网络流形，用PHATE降维可视化
- 高性能网络在流形中聚类，且多维度特征一致
- 可指导超参调优与架构搜索，适合模型优化研究者
受流形假设启发——高维数据通常位于或接近低维流形——我们对神经网络空间应用流形学习。通过定义神经网络隐藏层表示之间的距离，将网络作为点构建流形，并使用非线性降维算法PHATE生成网络流形。我们通过类别分离、层次聚类结构、谱熵和拓扑结构等特征对流形进行表征。分析显示，高性能网络在流形中聚集，且在所有特征上呈现一致的嵌入模式。最后，我们展示了该方法在引导超参数优化和神经架构搜索中的实用性，可通过从流形采样实现高效搜索。
原文摘要 · Abstract (English)
Drawing motivation from the manifold hypothesis, which posits that most high-dimensional data lies on or near low-dimensional manifolds, we apply manifold learning to the space of neural networks. We learn manifolds where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. We characterize this manifold using features of the representation, including class separation, hierarchical cluster structure, spectral entropy, and topological structure. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns across all these features. Finally, we demonstrate the utility of this approach for guiding hyperparameter optimization and neural architecture search by sampling from the manifold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。