研究如何用少量数据高效适配多模态大模型,提升图像分类性能。
Adapting Multimodal Foundation Models for Few-Shot Learning: A Comprehensive Study on Contrastive Captioners

- 提出从零训练到低秩微调的多级适配策略
- 强数据增强在少样本下稳定微调效果,但会降低线性探测性能
- 融合对比损失的混合目标显著提升少样本分类表现
大规模多模态基础模型,尤其是对比式生成模型CoCa,通过统一对比对齐与生成式描述实现了领先性能。尽管零样本迁移能力已被广泛验证,但这类生成-对比混合模型在极端数据稀缺场景(少样本学习)下的适应机制仍不明确。现有研究主要聚焦双编码器架构如CLIP,缺乏对CoCa独特隐空间在参数高效微调(PEFT)下响应的理解。本文系统评估了将CoCa视觉主干用于少样本图像分类的多种策略,涵盖无训练混合原型构建到基于低秩适配(LoRA)的深度参数调整。研究发现:强数据增强虽损害线性探测性能,却是稳定LoRA微调的关键;融合监督对比(SupCon)损失的混合目标在不同样本量下均优于标准交叉熵;并揭示了训练配置对数据稀缺的敏感性,为正则化、秩和采样策略的扩展提供了实证参考。
原文摘要 · Abstract (English)
Large-scale multimodal foundation models, particularly Contrastive Captioners (CoCa), have achieved state-of-the-art results by unifying contrastive alignment with generative captioning. While zero-shot transfer capabilities are well-documented, the adaptation of these generative-contrastive hybrids to downstream tasks with extreme data scarcity (few-shot learning) remains under-explored. Existing literature predominantly focuses on dual-encoder architectures like CLIP, leaving a gap in understanding how CoCa's distinct latent space responds to parameter-efficient fine-tuning (PEFT). This paper presents a comprehensive empirical study on adapting the CoCa visual backbone for few-shot image classification. We systematically evaluate a hierarchy of strategies, ranging from training-free hybrid prototyping to deep parameter adaptation via Low-Rank Adaptation (LoRA). First, we identify an "augmentation divergence": while strong data augmentation degrades the performance of linear probing in low-shot settings, it is essential for stabilizing LoRA fine-tuning. We also demonstrate that hybrid objectives incorporating Supervised Contrastive (SupCon) loss yield consistent performance improvements over standard Cross-Entropy across varying shot counts. Crucially, we characterize the sensitivity of training configurations to data scarcity, providing empirical reference settings for scaling regularization, rank, and sampling strategies to facilitate the efficient adaptation of generative-contrastive foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。