通过分组激活共享参数,实现高效微调与抗噪声推理。
GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers
- 将注意力层输出分组,每组共享缩放和偏移参数,大幅减少可训练参数
- 在多个基准上性能媲美或超越现有方法,可训练参数量降低一个数量级
- 引入随机扰动建模硬件噪声,适合边缘设备上的低功耗部署
参数高效微调(PEFT)通过仅更新大型预训练模型中的一小部分参数,提供了一种可扩展的全模型适配替代方案。我们提出GRASP——分组激活共享参数化,将选定层的D维标记表示划分为K << D个组,并为每组学习一个共享的缩放和偏移向量。这种分组调制显著减少了可训练参数数量,同时保持模型学习任务特异性特征的能力。基于此框架,我们进一步提出StochGRASP,学习高斯分布作为对预训练权重的扰动而非确定值。结合噪声感知损失函数,该概率参数化能建模硬件级权重变化,在非理想推理条件下显著提升鲁棒性,这对边缘新兴AI硬件部署至关重要。在GLUE(RoBERTa-base & RoBERTa-large)和E2E NLG(GPT-2 Medium)上,GRASP性能匹配或超过主流PEFT方法,且相比LoRA和BitFit可训练参数减少一个数量级。在不同噪声水平下,StochGRASP始终优于确定性变体,表明其适用于节能且易受噪声影响的硬件平台。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) provides a scalable alternative to full-model adaptation by updating only a small subset of parameters in large pre-trained models. We introduce GRASP - GRouped Activation Shared Parameterization - a lightweight PEFT framework that partitions the D-dimensional token representations of selected layers into K << D groups and learns a shared scaling and shifting vector for each group. This grouped modulation reduces the number of trainable parameters significantly while preserving the ability of the model to learn task-specific features. Building on this formulation, we further propose StochGRASP, which learns Gaussian distributions as perturbations to the pre-trained weights rather than deterministic values. This probabilistic parameterization along with a noise-aware loss function formulation enables modelling hardware-level variability in programmed weights and significantly improves robustness under non-ideal inference conditions-an important requirement for deployment on edge-based emerging AI hardware. Across GLUE (RoBERTa-base & RoBERTa-large) and E2E NLG (GPT-2 Medium), GRASP matches or exceeds the performance of established PEFT methods while achieving an order of magnitude reduction in trainable parameters compared to LoRA and BitFit. Under varying levels of noise, StochGRASP consistently outperforms deterministic variants, demonstrating its suitability for energy-efficient and noise-prone hardware platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。