只调模型特定参数，就能让CLIP在少样本下表现大幅提升。
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
- 仅微调偏置项和归一化层，无需增加额外参数。
- 零样本CLIP平均调和精度提升7.27%。
- 适合想高效改进视觉语言模型的开发者。
近期视觉语言模型（VLM）微调研究中，提示调优和适配器调优取得成功，而经典参数微调常被忽视。传统观点认为，使用少样本微调会破坏预训练知识，甚至导致性能下降。本文重新审视这一观点，提出：仅微调特定参数可释放经典微调的潜力。我们提出ClipFit方法，通过仅调整偏置项和归一化层，在不引入额外参数的情况下显著提升性能。实验表明，该方法使零样本CLIP的平均调和精度提升7.27%。进一步分析发现，低层级文本偏置层和第一层归一化层变化最明显。代码已开源。
原文摘要 · Abstract (English)
Recent advances in fine-tuning Vision-Language Models (VLMs) have witnessed the success of prompt tuning and adapter tuning, while the classic model fine-tuning on inherent parameters seems to be overlooked. It is believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. In this paper, we revisit this viewpoint, and propose a new perspective: fine-tuning the specific parameters instead of all will uncover the power of classic model fine-tuning on VLMs. Through our meticulous study, we propose ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27\% average harmonic mean accuracy. Lastly, to understand how fine-tuning in CLIPFit affects the pre-trained models, we conducted extensive experimental analyses w.r.t. changes in internal parameters and representations. We found that low-level text bias layers and the first layer normalization layer change much more than other layers. The code is available at \url{https://github.com/minglllli/CLIPFit}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。