arXiv:2603.10090cs.LG2026-03综述被引 13

探索神经网络权重空间的结构与生成,为模型分析和迁移提供新视角。

A Survey of Weight Space Learning: Understanding, Representation, and Generation

  • 将权重视为可分析的结构化空间,研究其几何与对称性。
  • 提出三类方法:理解、表示与生成权重空间,支持模型检索与知识迁移。
  • 适合关注模型分析、迁移学习与生成式建模的研究者参考。

神经网络权重通常被视为训练的最终产物,而当前深度学习研究多聚焦于数据、特征与架构。然而,近期进展表明,所有可能权重值的集合(权重空间)本身蕴含丰富结构:预训练模型形成有组织的分布,表现出对称性,并可嵌入、比较甚至生成。理解这些结构对模型分析、比较及知识跨模型迁移具有深远影响。这一新兴方向称为权重空间学习(WSL),将神经网络权重视为可分析与建模的有意义领域。本综述首次提出WSL的统一分类体系,将其方法分为三大核心维度:权重空间理解(WSU),研究权重的几何与对称性;权重空间表示(WSR),学习模型权重的嵌入;权重空间生成(WSG),通过超网络或生成模型合成新权重。我们进一步展示这些进展如何推动实际应用,包括模型检索、持续学习、联邦学习、神经架构搜索与无数据重建。通过整合分散成果,本综述凸显权重空间作为可学习、结构性域的潜力,在模型分析、迁移与生成中日益重要。配套资源已发布于 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning。

原文摘要 · Abstract (English)

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

权重空间模型分析生成模型迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。