提出新优化器SAMPLe,让视觉语言模型的提示学习更通用、不易过拟合。
SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs
- 基于损失曲面平坦度设计优化器,动态调节训练过程
- 在多个提示学习框架上提升性能,尤其改善未见数据表现
- 无需修改模型,适配性强,适合追求泛化能力的研究者
预训练视觉-语言模型(如CLIP)在众多下游任务中表现出色。然而,提示学习面临性能与泛化之间的权衡:尽管可通过调优提示获得高精度,但常导致对未见数据泛化能力下降。由于可学习提示数量有限,容易过拟合训练数据。为此,我们提出SAMPLe(Sharpness-Aware Minimization Prompt Learning),一种即插即用的尖锐性感知优化器,通过考虑损失曲面的尖锐度来增强提示的泛化能力。与传统方法不同,SAMPLe在每一步满足目标函数约束,依据局部曲率和梯度特性动态调整优化状态,平衡探索与利用。该方法有效减少对已见分布的过拟合,提升对未知数据的适应性,保留预训练模型的泛化潜力。我们将SAMPLe集成至CoOp、CoCoOp、MaPLe、TCP和Co-Prompt等多种提示学习框架,在多种设置下验证其有效性,结果表明它显著优于现有优化器,是一种鲁棒且模型无关的提示学习解决方案。
原文摘要 · Abstract (English)
Pre-trained Vision-Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for prompt learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。