用指令生成神经网络参数,让模型快速适配新任务。
Instruction-Guided Autoregressive Neural Network Parameter Generation
- 基于自回归机制,按令牌逐层生成参数以保证层间一致性。
- 在多个视觉数据集上表现优于或媲美现有方法,尤其擅长大模型。
- 适合需要快速定制模型的开发者,支持跨任务和架构灵活适配。
学习根据任务描述和架构规格生成神经网络参数,对提升模型适应性和迁移学习能力至关重要。现有方法,尤其是基于扩散模型的方法,在处理大型架构时扩展性差、难以应对不同深度的网络,且参数生成过程缺乏层间连贯性。本文提出IGPG(Instruction Guided Parameter Generation),一种统一多种任务与架构的自回归框架。IGPG利用VQ-VAE与自回归模型,根据任务指令、数据集及架构细节生成神经网络参数。通过自回归生成权重令牌,IGPG确保层间一致性,并实现模型与数据集间的高效适应。在令牌级别操作使其能有效捕捉来自广泛预训练模型的复杂参数分布。在多个视觉数据集上的大量实验表明,IGPG将多样化的预训练模型整合为单一灵活的生成框架。合成参数在性能上达到或超越当前最优方法,尤其在应用于大型架构时展现出卓越的可扩展性与效率。这些结果凸显了IGPG在预训练权重检索、模型选择与快速任务微调中的潜力。
原文摘要 · Abstract (English)
Learning to generate neural network parameters conditioned on task descriptions and architecture specifications is pivotal for advancing model adaptability and transfer learning. Existing methods especially those based on diffusion models suffer from limited scalability to large architectures, rigidity in handling varying network depths, and disjointed parameter generation that undermines inter-layer coherence. In this work, we propose IGPG (Instruction Guided Parameter Generation), an autoregressive framework that unifies parameter synthesis across diverse tasks and architectures. IGPG leverages a VQ-VAE and an autoregressive model to generate neural network parameters, conditioned on task instructions, dataset, and architecture details. By autoregressively generating neural network weights' tokens, IGPG ensures inter-layer coherence and enables efficient adaptation across models and datasets. Operating at the token level, IGPG effectively captures complex parameter distributions aggregated from a broad spectrum of pretrained models. Extensive experiments on multiple vision datasets demonstrate that IGPG consolidates diverse pretrained models into a single, flexible generative framework. The synthesized parameters achieve competitive or superior performance relative to state-of-the-art methods, especially in terms of scalability and efficiency when applied to large architectures. These results underscore ICPG potential as a powerful tool for pretrained weight retrieval, model selection, and rapid task-specific fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。