让CLIP模型用少量样本快速适应新任务，不需重新训练。
Attn-Adapter: Attention Is All You Need for Online Few-shot Learner of Vision-Language Model
- 通过双注意力机制动态调整图像和类别特征
- 在少样本场景下超越现有方法，跨数据集表现更优
- 适合需要快速适配新类别的视觉语言应用
对比学习的视觉-语言模型在零样本图像识别中表现优异，但在少样本场景下受限于计算量大、需离线微调的提示学习方法，易过拟合。为此，我们提出Attn-Adapter，一种新型在线少样本学习框架，通过双注意力机制提升CLIP的适应能力。设计包含两个组件：记忆注意力适配器（Memory Attn-Adapter）利用支持样本优化类别嵌入，局部-全局注意力适配器（Local-Global Attn-Adapter）融合局部与全局特征增强图像嵌入。该架构无需重训练基础模型，即可从少量标注样本实现动态适配。Attn-Adapter在跨类别与跨数据集泛化上优于当前最优方法，推理高效，并可扩展至不同规模的CLIP骨干网络。
原文摘要 · Abstract (English)
Contrastive vision-language models excel in zero-shot image recognition but face challenges in few-shot scenarios due to computationally intensive offline fine-tuning using prompt learning, which risks overfitting. To overcome these limitations, we propose Attn-Adapter, a novel online few-shot learning framework that enhances CLIP's adaptability via a dual attention mechanism. Our design incorporates dataset-specific information through two components: the Memory Attn-Adapter, which refines category embeddings using support examples, and the Local-Global Attn-Adapter, which enriches image embeddings by integrating local and global features. This architecture enables dynamic adaptation from a few labeled samples without retraining the base model. Attn-Adapter outperforms state-of-the-art methods in cross-category and cross-dataset generalization, maintaining efficient inference and scaling across CLIP backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。