用图像自增强提升视觉语言模型提示调优效果
Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
- 仅用原始数据内部增强，不依赖外部知识
- 通过共识测试门控机制过滤噪声样本，提升增强质量
- 无需额外成本，显著提升模型性能与泛化能力
基于CLIP的提示调优中，引入更多数据作为额外知识以增强微调过程已被证明是有效方法。现有提示调优的数据增强策略通常依赖外部知识（如大语言模型或预结构化知识库），导致数据收集与处理成本较高，且普遍忽略图像模态中已有的特征利用。为此，我们提出增强驱动的提示调优（AugPT），一种仅使用原始数据内部增强的自包含蒸馏式提示调优方法，以更好挖掘已有特征。具体而言，AugPT对训练集中未标注图像进行自监督增强，并引入基于共识测试的新门控机制，复用预训练的提示调优主干模型，自发过滤噪声样本，进一步提升增强视图的质量。大量实验验证，AugPT在不使用附加外部知识的情况下，同时提升了模型性能与泛化能力。代码已开源：https://github.com/JREion/AugPT。
原文摘要 · Abstract (English)
For CLIP-based prompt tuning, introducing more data as additional knowledge for enhancing fine-tuning process is proved to be an effective approach. Existing data amplification strategies for prompt tuning typically rely on external knowledge (e.g., large language models or pre-structured knowledge bases), resulting in higher costs for data collection and processing, while generally ignoring further utilization of features in image modality. To address this, we propose Augmentation-driven Prompt Tuning (AugPT), a self-contained distillation-based prompt tuning approach using only internal augmentation on raw dataset to better exploit known features. Specifically, AugPT employs self-supervised augmentation on unlabeled images in the training set, and introduces a novel gating mechanism based on consensus test, reusing the pre-trained prompt tuning backbone model to spontaneously filter noisy samples, further enhancing the quality of augmented views. Extensive experiments validate that AugPT simultaneously enhances model performance and generalization capability without using appended external knowledge. The code of AugPT is available at: https://github.com/JREion/AugPT .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。