用图像掩码提升视觉语言模型的提示学习泛化能力
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
- 通过掩码图像建模生成更鲁棒的条件提示
- 在零样本和少样本任务中显著提升泛化性能
- 无需修改原架构，计算开销极低，适合实际应用
视觉语言模型（如 CLIP）在零样本学习中表现优异，但适应新任务时通常需要高资源训练。提示学习方法（如 CoOp、CoCoOp）虽高效，却易过拟合已知类别，限制对未见类别的泛化能力。本文提出 ProMIM，一种即插即用框架，将掩码图像建模（MIM）融入现有视觉语言模型流程。ProMIM 采用简单有效的掩码策略，仅遮盖可见图像块，并利用其表征引导提示生成，从而增强特征鲁棒性、缓解过拟合，且引入可忽略的额外计算成本。在零样本与少样本分类任务上的广泛实验表明，ProMIM 能持续提升现有方法的泛化性能，为实际视觉语言应用提供轻量级、实用的解决方案。
原文摘要 · Abstract (English)
Vision-language models (VLMs) like CLIP excel in zero-shot learning but often require resource-intensive training to adapt to new tasks. Prompt learning techniques, such as CoOp and CoCoOp, offer efficient adaptation but tend to overfit to known classes, limiting generalization to unseen categories. We introduce ProMIM, a plug-and-play framework that enhances conditional prompt learning by integrating masked image modeling (MIM) into existing VLM pipelines. ProMIM leverages a simple yet effective masking strategy to generate robust, instance-conditioned prompts, seamlessly augmenting methods like CoOp and CoCoOp without altering their core architectures. By masking only visible image patches and using these representations to guide prompt generation, ProMIM improves feature robustness and mitigates overfitting, all while introducing negligible additional computational cost. Extensive experiments across zero-shot and few-shot classification tasks demonstrate that ProMIM consistently boosts generalization performance when plugged into existing approaches, providing a practical, lightweight solution for real-world vision-language applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。