arXiv:2510.02096cs.LG2025-10被引 4

用公开模型库训练权重空间表示,无需精心构建模型动物园。

Learning Model Representations Using Publicly Available Model Hubs

  • 从Hugging Face等杂乱模型库直接提取权重训练
  • 性能超越传统实验室生成的模型动物园
  • 可泛化到未见过的模型类型,适合开放场景研究

神经网络权重已成为一种新型数据模态,推动了权重空间学习的发展。该领域核心挑战在于,学习有意义的权重表征通常需要大规模、精心构建的已训练模型集合(即模型动物园),而这类集合往往需大量算力且难以扩展。本文提出一种新方法:直接在来自Hugging Face等大型非结构化模型仓库中下载的任意模型上训练权重空间学习主干网络。这些仓库包含架构和数据集高度异质的模型,且文档缺失。为此,我们设计了一种能处理无序模型群体的新权重空间主干。实验证明,基于Hugging Face模型训练的权重表征性能强劲,常优于在实验室生成的模型动物园上训练的模型。此外,训练集的多样性使我们的模型具备对未见数据模态的泛化能力。本工作表明,高质量的权重空间表示可在真实环境中学习,从而突破当前依赖人工构建模型动物园的瓶颈。

原文摘要 · Abstract (English)

The weights of neural networks have emerged as a novel data modality, giving rise to the field of weight space learning. A central challenge in this area is that learning meaningful representations of weights typically requires large, carefully constructed collections of trained models, typically referred to as model zoos. These model zoos are often trained ad-hoc, requiring large computational resources, constraining the learned weight space representations in scale and flexibility. In this work, we drop this requirement by training a weight space learning backbone on arbitrary models downloaded from large, unstructured model repositories such as Hugging Face. Unlike curated model zoos, these repositories contain highly heterogeneous models: they vary in architecture and dataset, and are largely undocumented. To address the methodological challenges posed by such heterogeneity, we propose a new weight space backbone designed to handle unstructured model populations. We demonstrate that weight space representations trained on models from Hugging Face achieve strong performance, often outperforming backbones trained on laboratory-generated model zoos. Finally, we show that the diversity of the model weights in our training set allows our weight space model to generalize to unseen data modalities. By demonstrating that high-quality weight space representations can be learned in the wild, we show that curated model zoos are not indispensable, thereby overcoming a strong limitation currently faced by the weight space learning community.

权重空间模型仓库泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。