用分组提示提升视觉语言模型抗偏差能力，仅调1.6‰参数就超越全量微调。
GroupCoOp: Group-robust Fine-tuning via Group Prompt Learning
- 为每类样本设计专属分组提示词，作为该组的语义代表
- 在5个数据集上均优于现有方法，少数群体识别率显著提升
- 适合需要轻量化且抗数据偏见的视觉任务部署
参数高效微调（PEFT）的视觉语言模型在各类视觉任务中表现优异，得益于其丰富的知识与泛化能力。然而，近期研究发现此类微调后的模型易受微调数据集中子组不平衡引发的虚假相关性影响。为此，我们提出群组鲁棒微调算法GroupCoOp，通过使用特定于子组的文本提示作为目标类别的多分类器，增强微调后模型的群组鲁棒性。借助视觉语言模型中文本编码器的丰富语义知识，即使在样本较少的子组上也能发现有效提示。利用各分类对应的群组提示，缓解了因子组不平衡导致的少数群体被忽略、嵌入空间分布散乱等问题。GroupCoOp在五个基准测试上对五种CLIP架构均取得最佳性能，且偶尔超越全网络微调方法，仅需训练网络参数的0.016%。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) of vision-language models (VLMs) excels in various vision tasks thanks to the rich knowledge and generalization ability of VLMs. However, recent studies revealed that such fine-tuned VLMs are vulnerable to spurious correlations stemming from the subgroup imbalance in the fine-tuning datasets. To resolve this issue, we propose Group Context Optimization (GroupCoOp), a simple and effective debiased fine-tuning algorithm that enhances the group robustness of fine-tuned VLMs. Its key idea is to employ group-specific text prompts as group representatives serving as multiple classifiers for their target class. The rich semantic knowledge of the text encoder of VLM enables the discovery of effective group prompts even for groups with a small number of training samples. Leveraging the group prompts for each class addresses the issues caused by the group-imbalanced training set, such as the neglect of minority groups and the scattered distribution of each class in the embedding space. GroupCoOp achieved the best results on five benchmarks across five CLIP architectures and occasionally outperformed prior methods that fine-tune the entire network, despite training only 0.016\% of the network's parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。