优化适配器位置可显著提升微调效率，少而精的放置比全层加适配器更好。
Towards Optimal Adapter Placement for Efficient Transfer Learning
- 扩展适配器连接方式，引入长距离与循环连接结构
- 随机选取的新位置在多个任务上表现优于每层都加适配器的基线
- 高性能位置常对应高梯度秩，适合追求高效微调的研究者
参数高效微调（PETL）旨在用最少的可训练参数将预训练模型适配到新下游任务。适配器是PETL中的主流方法，通过引入低秩投影增加网络容量，实现与全量微调相当的性能，同时参数量大幅减少。本文研究适配器位置与其性能之间的关系，发现其效果受网络内部位置影响显著，且最优位置依赖于具体任务。为此，我们提出一个扩展的适配器连接搜索空间，包含长距离和循环连接。实验表明，即使从该空间中随机选择适配器位置，也能获得更优结果；而表现优异的位置通常与高梯度秩相关。研究揭示：少量精心设计的适配器放置即可达到甚至超过每层添加适配器的基准性能，为最优适配器部署策略开辟了新方向。
原文摘要 · Abstract (English)
Parameter-efficient transfer learning (PETL) aims to adapt pre-trained models to new downstream tasks while minimizing the number of fine-tuned parameters. Adapters, a popular approach in PETL, inject additional capacity into existing networks by incorporating low-rank projections, achieving performance comparable to full fine-tuning with significantly fewer parameters. This paper investigates the relationship between the placement of an adapter and its performance. We observe that adapter location within a network significantly impacts its effectiveness, and that the optimal placement is task-dependent. To exploit this observation, we introduce an extended search space of adapter connections, including long-range and recurrent adapters. We demonstrate that even randomly selected adapter placements from this expanded space yield improved results, and that high-performing placements often correlate with high gradient rank. Our findings reveal that a small number of strategically placed adapters can match or exceed the performance of the common baseline of adding adapters in every block, opening a new avenue for research into optimal adapter placement strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。