让扩散模型能按需缩放,一次预训练即可适配不同大小设备。
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models
- 将模型权重分解为共享的‘学因’和层特定参数,实现知识可重组。
- 在有限数据上轻量微调后,新模型性能达当前最优水平。
- 适合需要快速部署多尺寸模型的场景,如移动端或边缘计算。
扩散模型训练成本高昂,有效预训练至关重要。但实际部署常需应对不同内存与算力约束,导致对应预训练模型缺失。为此,我们提出FINE方法,其生成的模型可将知识分解为称为‘learngenes’的基础组件,直接用于初始化多种尺寸模型,避免重复预训练。不同于传统全参数优化,FINE将每层权重表示为 $U_{ar{}}$、$Σ_{ar{}}^{(l)}$ 与 $V_{ar{}}^ op$ 的乘积,其中 $U_{ar{}}$ 和 $V_{ar{}}$ 为跨层共享的尺寸无关学因,$Σ_{ar{}}^{(l)}$ 保持层内特异性。联合训练这些组件后,形成可分解且可迁移的知识结构,通过灵活重组学因并仅对 $Σ_{ar{}}^{(l)}$ 在少量数据上轻量微调,即可高效初始化新模型。大量实验表明,FINE在资源受限场景下初始化变尺寸模型时表现优异,达到当前最佳性能;且由FINE初始化的模型能有效适应多种任务,体现学因的通用性。
原文摘要 · Abstract (English)
The training of diffusion models is computationally intensive, making effective pre-training essential. However, real-world deployments often demand models of variable sizes due to diverse memory and computational constraints, posing challenges when corresponding pre-trained versions are unavailable. To address this, we propose FINE, a novel pre-training method whose resulting model can flexibly factorize its knowledge into fundamental components, termed learngenes, enabling direct initialization of models of various sizes and eliminating the need for repeated pre-training. Rather than optimizing a conventional full-parameter model, FINE represents each layer's weights as the product of $U_{\star}$, $Σ_{\star}^{(l)}$, and $V_{\star}^\top$, where $U_{\star}$ and $V_{\star}$ serve as size-agnostic learngenes shared across layers, while $Σ_{\star}^{(l)}$ remains layer-specific. By jointly training these components, FINE forms a decomposable and transferable knowledge structure that allows efficient initialization through flexible recombination of learngenes, requiring only light retraining of $Σ_{\star}^{(l)}$ on limited data. Extensive experiments demonstrate the efficiency of FINE, achieving state-of-the-art performance in initializing variable-sized models across diverse resource-constrained deployments. Furthermore, models initialized by FINE effectively adapt to diverse tasks, showcasing the task-agnostic versatility of learngenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。