不改结构不加参数，仅调一个层就大幅提速提效
APLA: A Simple Adaptation Method for Vision Transformers
- 只更新注意力后投影层权重，无需修改架构或新增参数
- 在46个数据集上超越17种主流方法，最高省52.63%显存
- 适合追求高效微调的开发者，尤其在资源受限场景
现有适配技术通常需修改网络结构或引入额外参数，导致计算开销大、流程复杂。本文提出注意力投影层适配（APLA），一种无需改变视觉变换器（ViTs）架构且不增加参数的简单适配方法。通过系统分析发现，注意力机制后的第一层对适配至关重要。仅更新该投影层，甚至只需更新其随机子集权重，即可实现顶尖性能，同时显存占用最多降低52.63%，训练时间缩短43.0%，推理阶段无额外开销。在涵盖场景分类、医学影像、卫星图像和细粒度分类等46个不同任务的数据集上，APLA在分类、分割和检测任务中持续优于17种主流适配方法，包括全量微调。代码已开源：https://github.com/MoeinSorkhei/APLA。
原文摘要 · Abstract (English)
Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at https://github.com/MoeinSorkhei/APLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。