让视觉语言模型在少量数据下适应任务时,不丢失预训练知识。
Guided Prompt Evolution for Vision-Language Models Adaptation
- 用统一嵌入空间生成分层提示,控制提示演化路径。
- 分离方向与幅度更新,保留早期语义方向,仅调整强度。
- 适合需要保持原始能力的少样本场景,如医疗图像分析。
大规模视觉语言模型(VLMs)在少量标注数据下适应下游任务仍具挑战。尽管参数高效提示学习方法有潜力,但常导致预训练知识的灾难性遗忘。本文基于提示演化路径需受控的洞察,提出EvoPrompt框架,通过模态共享提示投影器(MPP)从统一嵌入空间生成分层提示。关键在于,进化式训练策略将低秩更新解耦为方向与幅度成分,保留早期学习的语义方向,仅调整其幅度,实现无遗忘的知识保持微调。大量实验表明,EvoPrompt在少样本学习中达到当前最优性能,同时稳健保持预训练VLM的零样本能力。
原文摘要 · Abstract (English)
The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge. While parameter-efficient prompt learning methods offer a promising path, they often suffer from catastrophic forgetting of pre-trained knowledge. Toward addressing this limitation, our work is grounded in the insight that governing the evolutionary path of prompts is essential for forgetting-free adaptation. To this end, we propose EvoPrompt, a novel framework designed to explicitly steer the prompt trajectory for knowledge-preserving fine-tuning. Specifically, our approach employs a Modality-Shared Prompt Projector (MPP) to generate hierarchical prompts from a unified embedding space. Critically, an evolutionary training strategy decouples low-rank updates into directional and magnitude components, preserving early-learned semantic directions while only adapting their magnitude, thus enabling prompts to evolve without discarding foundational knowledge. Extensive experiments demonstrate that EvoPrompt achieves state-of-the-art performance in few-shot learning while robustly preserving the original zero-shot capabilities of pre-trained VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。