通过捕捉视觉特征的分布关系，提升视觉语言模型在少样本下的表现。
Diversity Covariance-Aware Prompt Learning for Vision-Language Models
- 用马氏距离替代余弦距离，建模跨模态特征的协方差关系。
- 学习多个多样化软提示，实现多中心协方差建模，增强决策边界多样性。
- 在11个数据集上验证有效，适合少样本场景下的模型适配。
Prompt tuning 能在多种下游任务（如少样本学习）中进一步提升视觉语言模型性能，使其更适应具体应用需求。本文提出一种多样性协方差感知框架，通过学习数据分布信息来增强提示模型的少样本能力。首先，我们提出一种协方差感知方法，建模视觉特征间的协方差关系，并采用各向异性马氏距离替代次优的余弦距离，以衡量两模态间相似性，且严格推导并证明该建模过程的有效性。其次，提出多样性感知方法，学习多个多样化的软提示，捕捉类别不同属性，并独立对齐视觉模态，实现多中心协方差建模，从而生成更具多样性的决策边界。在11个不同任务的数据集上进行的大量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Prompt tuning can further enhance the performance of visual-language models across various downstream tasks (e.g., few-shot learning), enabling them to better adapt to specific applications and needs. In this paper, we present a Diversity Covariance-Aware framework that learns distributional information from the data to enhance the few-shot ability of the prompt model. First, we propose a covariance-aware method that models the covariance relationships between visual features and uses anisotropic Mahalanobis distance, instead of the suboptimal cosine distance, to measure the similarity between two modalities. We rigorously derive and prove the validity of this modeling process. Then, we propose the diversity-aware method, which learns multiple diverse soft prompts to capture different attributes of categories and aligns them independently with visual modalities. This method achieves multi-centered covariance modeling, leading to more diverse decision boundaries. Extensive experiments on 11 datasets in various tasks demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。