用视觉语言模型指导主动学习，少标注也能高效分类。
Active Prompt Learning with Vision-Language Model Priors
- 利用预训练模型的图文编码器做聚类，平衡采样
- 根据类别置信度自适应设定查询阈值，节省标注预算
- 在7个数据集上优于现有方法，适合标注成本高的场景
视觉语言模型（VLMs）在各类分类任务中展现出出色的零样本性能。然而，其对每个任务依赖人工设计的文本提示，限制了新任务的高效适配。尽管提示学习提供了解决方案，但多数研究聚焦于充分利用少量标注数据，常忽视精心的数据选择策略——该策略能在更少标注下实现更高准确率。为此，本文提出一种预算高效的主动提示学习框架。具体而言，引入类别引导聚类，利用VLM的预训练图像与文本编码器，在主动学习初始阶段生成聚类均衡的获取函数。同时，考虑到VLM在不同类别间置信度差异显著，提出基于自适应类别阈值的预算节省型选择性查询机制。在七个数据集上的主动学习实验表明，本方法显著优于现有基线。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have demonstrated remarkable zero-shot performance across various classification tasks. Nonetheless, their reliance on hand-crafted text prompts for each task hinders efficient adaptation to new tasks. While prompt learning offers a promising solution, most studies focus on maximizing the utilization of given few-shot labeled datasets, often overlooking the potential of careful data selection strategies, which enable higher accuracy with fewer labeled data. This motivates us to study a budget-efficient active prompt learning framework. Specifically, we introduce a class-guided clustering that leverages the pre-trained image and text encoders of VLMs, thereby enabling our cluster-balanced acquisition function from the initial round of active learning. Furthermore, considering the substantial class-wise variance in confidence exhibited by VLMs, we propose a budget-saving selective querying based on adaptive class-wise thresholds. Extensive experiments in active learning scenarios across seven datasets demonstrate that our method outperforms existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。