提出新方法让视觉语言模型微调模块在主模型更新后仍有效
Towards Compatible Fine-tuning for Vision-Language Model Updates
- 用注意力层融合可学习提示与类别嵌入，实现动态适配
- 15个数据集实验表明兼容性优于基线方法，泛化能力更强
- 适合需要持续更新的模型部署场景，如在线学习系统
目前高效微调已成为通过学习即插即用模块来增强基础模型下游任务性能的流行策略。然而，现有方法忽视了一个关键问题：若底层基础模型更新，这些即插即用模块是否仍有效？本文首次针对CLIP模型详细分析了多种微调方法在模型更新下的兼容性。研究发现，许多高性能微调方法在模型升级后失效。为此，我们提出一种新方法——类条件上下文优化（ContCoOp），通过注意力层将可学习提示与类别嵌入融合后再输入文本编码器。该设计使提示能动态适应嵌入空间变化（由模型更新引起），确保持续有效性。在15个数据集上的大量实验表明，ContCoOp在兼容性上优于基线方法，并展现出强泛化能力。
原文摘要 · Abstract (English)
So far, efficient fine-tuning has become a popular strategy for enhancing the capabilities of foundation models on downstream tasks by learning plug-and-play modules. However, existing methods overlook a crucial issue: if the underlying foundation model is updated, are these plug-and-play modules still effective? In this paper, we first conduct a detailed analysis of various fine-tuning methods on the CLIP in terms of their compatibility with model updates. The study reveals that many high-performing fine-tuning methods fail to be compatible with the upgraded models. To address this, we propose a novel approach, Class-conditioned Context Optimization (ContCoOp), which integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder. Consequently, the prompts can dynamically adapt to the changes in embedding space (due to model updates), ensuring continued effectiveness. Extensive experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。