构建了12个覆盖神经网络相变的可控模型库,助力权重空间学习研究。
A Model Zoo on Phase Transitions in Neural Networks
- 基于神经网络相变理论,系统构造12个结构化模型库。
- 涵盖视觉、语言、科学计算等多模态数据,覆盖所有已知相。
- 可用于训练分析、权重平均等下游任务,推动权重空间学习发展。
将训练好的神经网络模型权重作为数据模态,正成为新兴研究方向——权重空间学习(WSL)。现有模型库缺乏结构化和多样性定义。同时,统计物理研究发现神经网络存在相与相变,同一相内模型同质,跨相则性质迥异。本文结合相变理论与模型库概念,构建12个大规模、结构化的模型动物园,系统覆盖已知相,并在模型架构、规模与数据集上实现变化。数据涵盖计算机视觉、自然语言处理与科学机器学习等多模态任务。对每个模型计算损失景观指标,验证相的完整覆盖。该数据集为权重空间学习及更广泛应用提供资源。初步证据表明,损失景观相影响模型训练、分析与稀疏化等任务。我们通过迁移学习与权重平均的探索性研究,验证其有效性。
原文摘要 · Abstract (English)
Using the weights of trained Neural Network (NN) models as data modality has recently gained traction as a research field - dubbed Weight Space Learning (WSL). Multiple recent works propose WSL methods to analyze models, evaluate methods, or synthesize weights. Weight space learning methods require populations of trained models as datasets for development and evaluation. However, existing collections of models - called `model zoos' - are unstructured or follow a rudimentary definition of diversity. In parallel, work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another. We combine the idea of `model zoos' with phase information to create a controlled notion of diversity in populations. We introduce 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets. These datasets cover different modalities, such as computer vision, natural language processing, and scientific ML. For every model, we compute loss landscape metrics and validate full coverage of the phases. With this dataset, we provide the community with a resource with a wide range of potential applications for WSL and beyond. Evidence suggests the loss landscape phase plays a role in applications such as model training, analysis, or sparsification. We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。