用剪枝思路选微调位置，实现高效低秩更新。
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
- 基于激活加权重要性筛选固定稀疏参数更新区域。
- 在数学推理任务中，平均准确率超越多数适配器配置。
- 适合资源受限场景下的模型微调，尤其配合低秩适配器使用。
大型语言模型微调成本高昂，因全参数更新需大量内存、算力和任务存储。本文研究是否可复用原用于模型剪枝的显著性信号来决定模型应在哪部分进行适应。提出 Super 方法：通过一次校准遍历计算 Wanda 风格的激活加权重要性得分（Sun et al., 2023），固定一小部分可训练支持参数。进一步引入 Supra，一种结合该稀疏更新与 LoRA 的混合适配器，通过简单的预算分割规则保持匹配的可训练参数量。在 Llama-3.2-1B 与 Meta-Llama-3-8B 上的单种子 Math17K 算术实验表明，最优的 Super/Supra 变体在所有测试调度选择的适配器配置中取得最高平均准确率。同时引入类似 PaFi 的仅基于幅度的无训练稀疏基线，发现无论按幅度或 Wanda 式排序，低分支持区域仍具有效性。结果表明，简单的剪枝启发式排序可为参数高效微调提供有效固定稀疏支持，尤其与低秩适配器结合时表现更佳。
原文摘要 · Abstract (English)
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。