通过多语义引导优化提示,提升小样本学习泛化能力
MSGCoOp: Multiple Semantic-Guided Context Optimization for Few-Shot Learning
- 用多个可学习上下文向量捕捉不同语义特征
- 在11个数据集上平均提升1.10%的基类到新类泛化性能
- 无需复杂结构,适合资源受限场景使用
视觉-语言预训练模型(如CLIP)展现出强大的零样本泛化能力,提示学习成为替代全微调的有效方法。然而,现有方法常因过度拟合已见类别而难以泛化到新类别,且部分改进方法引入复杂架构或高计算开销。本文提出多语义引导上下文优化(MSGCoOp)框架,在保持高效的同时增强少样本泛化能力。该方法采用并行可学习上下文向量的集成,通过大语言模型自动生成的全面类别描述进行语义引导,丰富提示内容;同时引入多样性正则化损失,促使各提示学习互补且正交的特征,避免冗余。在11个基准数据集上的大量实验表明,MSGCoOp显著提升了基类到新类的泛化性能,平均调和均值优于强基线KgCoOp 1.10%。该方法在跨域泛化任务中也表现出更强鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Vision-language pre-trained models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, and prompt learning has emerged as an efficient alternative to full fine-tuning. However, existing methods often struggle with generalization to novel classes, a phenomenon attributed to overfitting on seen classes and forgetting general knowledge. Furthermore, recent approaches that improve generalization often introduce complex architectures or heavy computational overhead. In this paper, we propose a Multiple Semantic-Guided Context Optimization (MSGCoOp) framework to enhance few-shot generalization while maintaining computational efficiency. Our approach leverages an ensemble of parallel learnable context vectors to capture diverse semantic aspects. To enrich these prompts, we introduce a semantic guidance mechanism that aligns them with comprehensive class descriptions automatically generated by a Large Language Model (LLM). Furthermore, a diversity regularization loss encourages the prompts to learn complementary and orthogonal features, preventing them from collapsing into redundant representations. Extensive experiments on 11 benchmark datasets show that MSGCoOp significantly improves performance on base-to-novel generalization, achieving an average harmonic mean improvement of 1.10\% over the strong KgCoOp baseline. Our method also demonstrates enhanced robustness in cross-domain generalization tasks. Our code is avaliable at: \href{https://github.com/Rain-Bus/MSGCoOp}{https://github.com/Rain-Bus/MSGCoOp}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。