零初始化低秩适配器让冻结视觉模型更稳定，精度提升显著。
AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers
- 采用零初始化的低秩残差瓶颈，防止特征漂移。
- 在9个数据集上平均提升14.9%准确率，仅需0.92%参数量训练。
- 适合资源受限场景，优化稳定且可解释性强。
冻结主干的视觉变换器在迁移学习中面临两个未被充分解决的问题：将适配器直接插入固定特征提取器时优化不稳，以及缺乏设定适配器容量的合理指导。我们提出AdapterTune，为每个Transformer块引入一个残差低秩瓶颈，其上投影层零初始化，确保适配网络初始时恰好等于预训练函数，消除早期阶段的表示漂移。理论上，我们将适配器秩视为特征空间中逼近下游任务变化的容量预算。由此得到的过拟合风险分解预测了精度随秩增加而单调上升但逐渐减弱，呈现‘拐点’行为，经受控实验验证。我们在9个数据集和3种主干规模上进行评估，全程多种子报告。在核心5个数据集的迁移套件中，AdapterTune相较于仅调头部的方法平均提升14.9个百分点，训练参数量仅为全微调所需0.92%，并在15对数据集-主干组合中的10对上超越全微调。在整个基准测试中，所有数据集-主干组合均优于仅调头部。关于秩、位置和初始化的消融实验验证了各设计选择的有效性。代码已公开于：https://github.com/salimkhazem/adaptertune
原文摘要 · Abstract (English)
Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。