用简单正则化方法让CLIP模型在少样本分类上表现更优
CLIP's Visual Embedding Projector is a Few-shot Cornucopia
- 通过正则化投影矩阵微调视觉编码器，无需改变模型结构
- 在11个少样本基准上达到当前最优，训练速度比提示调优快十倍
- 适用于无法访问权重的黑箱场景，且对学习率不敏感
我们提出ProLIP，一种简单且与架构无关的方法，用于将对比预训练的视觉-语言模型（如CLIP）适配到少样本分类任务。ProLIP通过在投影矩阵与预训练权重偏差上施加弗罗贝尼乌斯范数正则化来微调视觉编码器。该方法在11个少样本分类基准上，无论是在“少样本验证”还是“无验证”设置下，均取得当前最优性能。此外，通过重新审视非线性CLIP-Adapter，我们设计出正则化线性适配器（RLA），其表现更优，无需超参数调优，对学习率不敏感，并可在无法访问模型权重的黑箱场景中替代ProLIP。除少样本分类外，ProLIP在跨数据集迁移、领域泛化、基础类到新类泛化及测试时适应等任务中也显著优于提示调优，且训练速度快一个数量级。代码已开源：https://github.com/astra-vision/ProLIP。
原文摘要 · Abstract (English)
We introduce ProLIP, a simple and architecture-agnostic method for adapting contrastively pretrained vision-language models, such as CLIP, to few-shot classification. ProLIP fine-tunes the vision encoder's projection matrix with Frobenius norm regularization on its deviation from the pretrained weights. It achieves state-of-the-art performance on 11 few-shot classification benchmarks under both ``few-shot validation'' and ``validation-free'' settings. Moreover, by rethinking the non-linear CLIP-Adapter through ProLIP's lens, we design a Regularized Linear Adapter (RLA) that performs better, requires no hyperparameter tuning, is less sensitive to learning rate values, and offers an alternative to ProLIP in black-box scenarios where model weights are inaccessible. Beyond few-shot classification, ProLIP excels in cross-dataset transfer, domain generalization, base-to-new class generalization, and test-time adaptation--where it outperforms prompt tuning while being an order of magnitude faster to train. Code is available at https://github.com/astra-vision/ProLIP .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。