让微调更高效：用智能模块动态生成更新，无需全量参数调整
PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers
- 引入权重感知的神经微调模块，根据预训练权重动态生成更新
- 在20多个数据集上超越主流方法，性能提升显著且开销低
- 适合资源受限场景下的模型适配，尤其适用于大模型轻量微调
大规模预训练模型微调虽能取得优异下游性能，但全参数更新成本过高。参数高效微调（PEFT）方法如LoRA通过引入轻量级更新模块缓解此问题，但通常依赖无权重感知的线性近似，表达能力受限。本文提出PEANuT，一种新型PEFT框架，引入权重感知的神经微调模块，可基于冻结的预训练权重生成任务自适应的更新。该方法在保持极低计算开销的同时，灵活捕捉复杂更新模式。理论分析表明，PEANuT在参数量相当或更少的情况下，表达能力等价或优于现有线性PEFT方法。在涵盖四类基准、超过二十个数据集的广泛实验中，PEANuT在自然语言处理与视觉任务上持续优于强基线。
原文摘要 · Abstract (English)
Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。