不用提取描述,直接蒸馏LLM知识到提示词中,提升视觉识别性能。
Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation
- 跳过提取文本描述,直接将LLM知识蒸馏到连续提示向量中
- 在11个识别数据集上表现优于现有方法,尤其在未见数据上更鲁棒
- 适合需要少标注、强泛化能力的视觉任务研究者
预训练视觉语言模型(VLM)通过提示学习可有效适配下游任务,无需额外标注配对数据。为补充VLM在视觉关联训练中缺失的文本信息,现有方法利用大语言模型(LLM)生成描述性文本融入提示,但存在响应变异大、可靠性低的问题。本文提出无描述多提示学习(DeMul),摒弃描述提取步骤,直接将LLM知识蒸馏至提示中。该方法使提示保留更丰富语义,同时以连续向量形式表示,支持优化且无需离散模板。在多提示设置中,我们实证验证了提示加权能反映不同提示的重要性。实验表明,该方法在11个识别数据集上均取得更优性能。
原文摘要 · Abstract (English)
Recent advances in pre-trained Vision Language Models (VLM) have shown promising potential for effectively adapting to downstream tasks through prompt learning, without the need for additional annotated paired datasets. To supplement the text information in VLM trained on correlations with vision data, new approaches leveraging Large Language Models (LLM) in prompts have been proposed, enhancing robustness to unseen and diverse data. Existing methods typically extract text-based responses (i.e., descriptions) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliability. In this work, we propose Description-free Multi-prompt Learning(DeMul), a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts. By adopting a description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates. Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training. Experimental results show that our approach achieves superior performance across 11 recognition datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。