按参数尺度动态调整更新，用极少参数实现更优下游适应
GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
- 根据参数初始值比例决定更新强度，避免无效微调
- 仅更新0.1%参数即达全量微调效果，最高提升1.6%准确率
- 自适应每层更新数量，适合资源有限的模型部署
参数高效微调（PEFT）已成为将大模型适配新任务的主流方法。现有方法仅更新少量参数而冻结其余部分，但常忽略参数原始尺度，导致行为变化微弱。本文提出梯度-权重比与熵引导掩码（GEM），一种尺度感知、分布敏感的稀疏微调框架。GEM优先更新相对于初始值变化显著的参数，并基于参数值的熵动态决定每层更新数量，最大化计算预算效率。实证表明，GEM在通用任务（GLUE、SuperGLUE）和领域特定任务（GSM8k、MBPP）上均有效，仅更新0.1%参数即可达到全量微调精度，最高提升1.6%。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has become a popular way to adapt large pre-trained models to new tasks. Most PEFT methods update only a small subset of parameters while freezing the rest, avoiding redundant computation. As they maximize the absolute size of the updates without regard to the parameters' original scale, the resulting changes in model behavior can be minimal. In contrast, we maximize updates relative to each parameter's scale, yielding more meaningful downstream adaptation. We propose Gradient-to-Weight Ratio and Entropy-guided Masking (GEM), a parameter scale-aware, distribution-sensitive sparse fine-tuning framework. GEM prioritizes parameters whose updates are significant in proportion to their initial pre-trained values. It also adaptively determines how many parameters to tune at each layer based on the entropy of parameter values, thereby making the most effective use of the computational budget in PEFT. Our empirical study demonstrates the efficacy of GEM on both general-domain tasks (GLUE and SuperGLUE) and domain-specific tasks (GSM8k and MBPP), achieving up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。