通过剪枝与共享适配器，实现视觉大模型高效微调。
Pear: Pruning and Sharing Adapters in Visual Parameter-Efficient Fine-Tuning
- 剪掉冗余适配器，共享重要适配器以节省存储。
- 保留剪掉适配器的知识信息，性能不降反升。
- 适合资源受限下需要持续微调的视觉模型应用。
适配器被广泛用于缓解微调预训练视觉基础模型时的计算和存储开销。然而，适配器本身可能产生冗余，导致不必要的存储负担并影响性能。本文提出一种新的适配器剪枝框架 Pear，通过剪除部分适配器，并将更重要的未剪枝适配器共享给被剪枝的位置，实现剪枝后这些位置仍可继续适应。此外，引入知识检查点策略，保留被剪枝适配器的信息，进一步提升性能。在视觉适应基准上的实验验证了 Pear 在效率与效果上优于其他竞争方法。代码见 https://github.com/yibozhong/pear。
原文摘要 · Abstract (English)
Adapters have been widely explored to alleviate computational and storage costs when fine-tuning pretrained foundation models. However, the adapter itself can exhibit redundancy, leading to unnecessary storage overhead and inferior performance. In this paper, we propose Prune and Share (Pear), a novel adapter-pruning framework for efficient fine-tuning of pretrained visual foundation models. Specifically, we prune certain adapters and share the more important unpruned ones with positions where adapters are pruned, allowing continual adaptation at these positions after pruning. Additionally, a knowledge checkpoint strategy is introduced, which preserves the information of the pruned adapters and further boosts performance. Experimental results on visual adaptation benchmark validate the effectiveness and efficiency of the proposed Pear comparing to other competitive methods. Code is in https://github.com/yibozhong/pear.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。