用生成器动态调整预训练模型权重，提升微调效率与性能。
GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models
- 以预训练权重为条件生成任务专用更新参数
- 在多领域评测中表现优于或媲美现有方法
- 适合资源有限但需高效适配大模型的场景
参数高效微调（PEFT）通过学习少量任务特定更新ΔW，实现对预训练基础模型（PFM）的高效适配。现有方法大多将ΔW独立于预训练权重$W_0$学习，或仅通过初始化、简单重参数化利用$W_0$。为更充分挖掘$W_0$中的结构信息，本文提出生成式参数高效微调（GenFT），一种以$W_0$为条件的PEFT方法，使用确定性权重生成器生成任务相关更新。具体地，GenFT通过带非线性激活的行与列变换从$W_0$中提取结构模式，并引入共享-特定分解，平衡跨层信息复用与层间灵活性。该方法简洁且参数高效，在自然语言处理与计算机视觉多个基准测试中达到竞争力或更优平均表现。我们还对LLaMA-7B进行了初步研究，验证其在生成类模型上的可行性。代码已开源于GitHub：https://github.com/xuguangning1218/GenFT。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has emerged as a resource-efficient strategy for adapting Pretrained Foundation Models (PFMs) by learning a small number of task-specific updates $ΔW$. Existing methods often learn $ΔW$ largely independently of pretrained weights $W_0$, or exploit $W_0$ mainly through initialization or simple reparameterization. To further leverage the structural information encoded in $W_0$, we propose Generative Parameter-Efficient Fine-Tuning (GenFT), a $W_0$-conditioned PEFT method that uses a deterministic weight generator to produce task-specific updates. Specifically, GenFT performs row and column transformations with nonlinear activations to extract structured patterns from $W_0$, and introduces a shared-specific decomposition to balance cross-layer information reuse and layer-specific flexibility. GenFT is simple and parameter-efficient, achieving competitive or better average performance across NLP and CV benchmarks. We further provide a pilot study on LLaMA-7B to examine its feasibility for generative models. The code is available at GitHub https://github.com/xuguangning1218/GenFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。