通过频域分解实现零样本知识迁移,任意大小模型都能快速复用预训练知识。
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
- 用离散余弦变换提取权重低频成分,作为通用知识载体。
- 在视觉任务中提速15倍,语言任务减少40.5%计算量。
- 无需训练即可适配不同规模模型,适合资源受限场景。
大规模预训练模型微调已成为下游任务的标准范式,但其知识与单一架构紧密耦合,难以灵活跨模型尺度复用。现有方法或依赖参数选择(忽略结构关联),或使用生成模型预测参数(需大量网络集合支持)。本文发现模型权重的低频分量是基础性、任务无关知识的载体——“learngene”,并验证其可高效传递至下游模型与任务。基于此,提出FRONT框架:利用离散余弦变换(DCT)提取低频“learngene”,可通过简单截断或填充无缝初始化任意规模模型,全程免训练。为提升性能,引入低开销谱正则化进行优化。大量实验表明,FRONT达当前最优,视觉任务收敛加速最高15倍,语言任务平均减少40.5%训练浮点运算量。代码已开源。
原文摘要 · Abstract (English)
Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales. In response to this challenge, recent approaches typically resort to either parameter selection, which fails to capture the interdependent structure of this knowledge, or parameter prediction using generative models that depend on impractical access to large network collections. In this paper, we identify the low-frequency components of model weights as the concrete carrier of foundational, task-agnostic knowledge, its ``learngene", and validate this by demonstrating its efficient inheritance by downstream models and tasks. Based on this insight, we propose FRONT (FRequency dOmain kNowledge Transfer), a novel framework that uses the Discrete Cosine Transform (DCT) to isolate the low-frequency ``learngene". This learngene can be seamlessly adapted to initialize models of arbitrary size via simple truncation or padding, a process that is entirely training-free. For enhanced performance, we propose an optional low-cost refinement process that introduces a spectral regularizer to further improve the learngene's transferability. Extensive experiments demonstrate that FRONT achieves the state-of-the-art performance, accelerates convergence by up to $15\times$ in vision tasks, and reduces training FLOPs by an average of 40.5% in language tasks. Code is available at https://github.com/LUcy0505/FRONT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。