通过提升误分类样本的提示,显著增强视觉语言模型的少样本分类性能
AdaBoosting Text Prompts for Vision-Language Models

- 基于AdaBoost思想,逐轮优化错误样本对应的文本提示
- 在11个基准上提升源模型准确率,跨模型迁移效果更优
- 适合需要少样本适配且追求跨模型通用性的研究者
预训练视觉语言模型(VLM)的分类准确率依赖于文本提示的质量。手工设计模板和大语言模型生成的描述不仅使预测更具可解释性,还能在异构VLM间复用。近期工作利用少量标注图像构建任务自适应文本提示,但现有少样本文本提示方法未在构造过程中显式关注误分类样本,导致即使增加样本数量,性能提升也十分有限。为充分挖掘少样本监督潜力,我们提出文本提示提升(TPB),一种受AdaBoost启发的框架:将每个基于文本提示的分类器视为弱学习器,通过显式聚焦难例(误分类样本)逐步聚合为强集成模型。大量实验表明,TPB在文本空间中保留了任务内固有、模型无关的线索,实现了稳健的跨模型迁移。在11个分类基准上,TPB提升了源模型的准确率,并在迁移到更大、更强的VLM时保持了随样本数增长的收益,而现有方法难以维持此类提升。
原文摘要 · Abstract (English)
The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。