通过动态丢弃视觉语言模型中的关键令牌，提升模型在少样本等挑战场景下的鲁棒性。
Dropout Prompt Learning: Towards Robust and Adaptive Vision-Language Models
- 基于跨模态上下文与对齐关系，为每个令牌动态设定丢弃概率。
- 在15个基准上验证，显著优于现有正则化方法，尤其在长尾分布和域外泛化中表现突出。
- 适合需要高鲁棒性的视觉语言模型部署，如开放世界识别任务。
Dropout是一种广泛应用的正则化技术，通过随机丢弃神经元提升模型泛化能力。本文提出一种新的Dropout Prompt Learning方法，旨在通过将dropout应用于视觉和文本分支的输入令牌，增强视觉-语言模型的鲁棒性。不同于传统dropout，该方法基于单模态上下文和跨模态对齐来评估每个令牌的重要性，从而实现对每个令牌灵活的丢弃概率。此外，为保持通用知识迁移的语义对齐同时激发由丢弃引入的多样化表示，我们进一步引入残差熵正则化。在15个基准上的实验表明，该方法在低样本学习、长尾分类及分布外泛化等挑战性场景下均表现出色。特别地，在基础到新类的泛化任务中，性能超越KgCoOp 5.10% 和PromptSRC 2.13%。
原文摘要 · Abstract (English)
Dropout is a widely used regularization technique which improves the generalization ability of a model by randomly dropping neurons. In light of this, we propose Dropout Prompt Learning, which aims for applying dropout to improve the robustness of the vision-language models. Different from the vanilla dropout, we apply dropout on the tokens of the textual and visual branches, where we evaluate the token significance considering both intra-modal context and inter-modal alignment, enabling flexible dropout probabilities for each token. Moreover, to maintain semantic alignment for general knowledge transfer while encouraging the diverse representations that dropout introduces, we further propose residual entropy regularization. Experiments on 15 benchmarks show our method's effectiveness in challenging scenarios like low-shot learning, long-tail classification, and out-of-distribution generalization. Notably, our method surpasses regularization-based methods including KgCoOp by 5.10% and PromptSRC by 2.13% in performance on base-to-novel generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。